akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Different init for training ensembles

rahulvigneswaran: <p>Currently am fixing the all seeds by doing the following,</p>
<pre><code class="lang-python">random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
np.random.seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
</code></pre>
<p>I want to train multiple models of the same architecture type but with different initialization and ensemble them later.</p>
<p>I usually change the seed to get a different init but since am fixing the seed for everything, how do I still get a different init?</p>

ptrblck: <p>Setting the seed at the beginning of the script would make the pseudorandom number generator output deterministic “random” values. Creating multiple models in the same script would thus also create different parameters, since the sequence of the random number generation is defined by the seed, but the values won’t be the same.</p>
Cross entropy for the batch with different masks

MrPositron: <p>Hello to everyone!</p>
<p>TLDR. I want to compute cross entropy for the batch, and in the batch there can be different masking strategy for each element in a batch.</p>
<p>I am having quite interesing question. Consider that my output returns a tensor of shape <code>[B, N, C]</code>. <code>B</code> is the number of batches, <code>N</code> is the sequence length and <code>C</code> is the length of vocabulary.</p>
<p>To compute cross entropy loss I want to get each 2nd word, or each 3rd word, etc. depending on the original sequence length. However, I am not sure whether it is possible to parallelize this operation.</p>

ptrblck: <p>You should be able to slice the model output and targets in the temporal dimension and only pass the desired time steps to the criterion. Did you try it out and if so did you see any issues?</p>
<p>Unrelated to this, but you would have to <code>permute</code> the model output so that the shape is <code>[batch_size, nb_classes, seq_len]</code>.</p>
BatchNorm1d‘s input is strange for me

kongmuqiuren: <p>If there is a 2d input, the second dimension represents Channel, it’s nature. But I cannot understand why is the same in 3d input?</p>
<pre><code class="lang-auto"># size:3*2 , 3 tensors, each have 2 channels
input=[[121,31,14][4,41,41]]

# size:2*3*2 , 2 batches, each batch contains 3 tensors, each have 2 channels
# why BatchNorm1d suggest the second dimension is channel dimension?
input=[[[121,31,14][4,41,41]],[[11,1,14][4,314,1]]]
</code></pre>

googlebot: <p>it is adapted for convolution networks, that use (Batch, Channels, *) convention, where * - 1-3 spatial dimensions.</p>

kongmuqiuren: <p>Thanks. I am not familiar with CV. So I wanna confirm do you mean that  for  RGB piectures, which size equals <code>batch*channel*height*width</code>，the <code>height*width</code> is the spatial dimension?</p>

googlebot: <p>yes, it is two spatial dimensions, usecase for 1d convolutions would be with time dimension.</p>
Does torchscript optimize models automatically to execute in parallel?

Chenchao_Zhao: <p>Suppose I have a model like this</p>
<pre><code class="lang-auto">class Model(nn.Module):
  def __init__(self):
    self.part1 = nn.Linear(10, 10)
    self.part2 = nn.Linear(10, 10)
  def forward(self, x):
    x1 = self.part1(x)
    x2 = self.part2(x)
   return x1 + x2
</code></pre>
<p>In eager mode, I’m sure it’s not executed in parallel but when I <code>torch.jit.script</code> the model, does torchscript automatically turn them into to async?<br>
If so does the optimization depend on device being cpu or cuda?</p>

SplitInfinity: <p>No, the ops are executed serially, but <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html" rel="noopener nofollow ugc">there may be intra-op parallelism</a>.</p>
AttributeError: 'Series' object has no attribute 'as_matrix'

SangYC: <p>When I execute the code of the official website, I get such an error. Why?<br>
code show as follow:</p>
<p>landmarks_frame = pd.read_csv(‘F:\OfficialData\faces\face_landmarks.csv’)</p>
<p>n = 65<br>
img_name = landmarks_frame.iloc[n, 0]<br>
landmarks = landmarks_frame.iloc[n, 1:].as_matrix()<br>
landmarks = landmarks.astype(‘float’).reshape(-1, 2)</p>
<p>print(‘Image name: {}’.format(img_name))<br>
print(‘Landmarks shape: {}’.format(landmarks.shape))<br>
print(‘First 4 Landmarks: {}’.format(landmarks[:4]))</p>

ptrblck: <p>Which line of code is raising this error and which pandas version are you using?<br>
I just tried to reproduce the issue, but the code works fine using <code>pandas==0.25.3</code>.</p>

SangYC: <p>“landmarks = landmarks_frame.iloc[n, 1:].as_matrix()”<br>
The above code runs with errors.</p>
<p>The version of pandas is 1.0.1.<br>
Replace ‘as_matrix()’  with  ‘to_numpy()’  and the problem is solved.</p>

ptrblck: <p>I used the code from the <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">tutorial</a>, which is using <code>np.asarray</code> and didn’t realize you’ve changed the code.<br>
Anyway, good to hear it’s working now.</p>

DJ007: <p>thanks, it works for me too</p>
For two models, how many optimizers do I need?

newtopytorch: <p>I currently have the following data:</p>
<blockquote>
<p>f_map, inputs, s_bias = ml_dataset.dataset_for_s_bias()</p>
</blockquote>
<p>where f_map is a tensor of matrices, inputs is a tensor of floats, and s_bias is a tensor of floats. The first two tensors, f_map and inputs, are the inputs to my ML regression algorithm, and s_bias is the expected output. The reason there are two kinds of inputs is because f_map is processed using a CNN, and the CNN turns each matrix in f_map into a float, which is concatenated with the inputs tensor, and the resulting tensor is inputted into an MLP to get a prediction for s_bias.</p>
<p>With this in mind, my model looks like this:</p>
<blockquote>
<p>def load_model(lr, n_filters, filter_sizes, spp_dim, input_size, hidden_size, output_size):<br>
cnn_model = CNN(n_filters, filter_sizes, spp_dim)<br>
mlp_model = MultiLayerPerceptron(input_size, hidden_size, output_size)<br>
loss_fnc = torch.nn.MSELoss()<br>
optimizer = torch.optim.SGD(mlp_model.parameters(), lr=lr)</p>
<pre><code>return cnn_model, mlp_model, optimizer, loss_fnc
</code></pre>
</blockquote>
<p>I didn’t know which model to use for my optimizer, so I just put the MLP model. However, I’m not sure if this is correct. Do I need a second optimizer for my CNN model? Or does the optimizer only need the final prediction, which comes out of the MLP model?</p>

ptrblck: <p>The optimizer would need the references to all parameters it should update. In the usual case you would either create two separate optimizers (one for each model) or pass the parameters of both models to a single optimizer.<br>
In your current setup you would also train <code>mlp_model</code> and <code>cnn_model</code> would be randomly initialized and not trained.</p>
How to get MNIST data from torchvision with three channels for some pretrained model like VGG?

deJQK: <p>I think this should be encountered by many people, and is not difficult, but when I want to use the MNIST data from torchvision with a model pretrained from torchvision.models, such as VGG, I got the error:</p>
<blockquote>
<p>Given groups=1, weight of size [64, 3, 3, 3], expected input[64, 1, 28, 28] to have 3 channels, but got 1 channels instead</p>
</blockquote>
<p>It seems that the model requires 3 channel inputs, but the data from torchvision is 1 channel, for MNIST.</p>

deJQK: <p>One ugly method is to add <code>transforms.Grayscale(3)</code> in the transform of the dataloader.</p>

ptrblck: <p>Another way would be to add a <code>nn.Conv2d</code> layer at the beginning of the pretrained model which gives you 3 output channels. However, you would need to train this layer.</p>

deJQK: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. I’m not sure how other framework like tensorflow treat this issue, but it seems to me some inconsistency of the package.</p>

Arshiz_K: <p>Hi, can you please explain how to do this for GoogleNet?</p>

ptrblck: <p>You could either replace the first layer in GoogleNet with e.g. an <code>nn.Sequential</code> container containing the new conv layer as well as the pretrained one:</p>
<pre><code class="lang-python">model = models.googlenet()
model.conv1 = nn.Sequential(
    nn.Conv2d(1, 3, 1),
    model.conv1)

x = torch.randn(1, 1, 224, 224)
out = model(x)
</code></pre>
<p>or just use an extra layer before the model:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 3, 1)
model = models.googlenet()

x = torch.randn(1, 1, 224, 224)
out = conv(x)
out = model(out)
</code></pre>
Grad is None for leaf tensor

ka86: <p>I am trying to compare the empirical distributions of elements of two tensors by computing a coarse histogram of the two tensors (<code>torch.histc</code> is not differentiable). I want to compute the gradients of a particular loss function measuring the dissimilarity between the two distributions w.r.t the input tensor which is a leaf variable. However the returned grad is always None. Below is a minimal example of what I’m trying to do. Any suggestions?</p>
<pre><code class="lang-auto">import torch

def hist_loss(X, Y, bins: int=10):
   
    X = X.view(-1)
    Y = Y.view(-1)
    xmin, xmax = X.min(), X.max()
    ymin, ymax = Y.min(), Y.max()
    xstep = (xmax - xmin)/bins
    ystep = (ymax - ymin)/bins
    hist_loss = torch.zeros(1, requires_grad=True).to(X)
    zero = torch.zeros(1, requires_grad=True).to(X)
    one = torch.ones(1, requires_grad=True).to(X)
 
    n = len(X)
    for i in range(bins):
        # x is the empirical distribution of the X
        # y is the empirical distribution of the Y
        x = torch.sum(torch.where(((X&gt;xmin+i*xstep) &amp; (X&lt;xmin+(i+1)*xstep)), one, zero))
        y = torch.sum(torch.where(((Y&gt;ymin+i*ystep) &amp; (Y&lt;ymin+(i+1)*ystep)), one, zero))
        hist_loss = hist_loss + ((x-y)/n)**2

    hist_loss    = torch.sqrt(hist_loss)

    return hist_loss

x = torch.randn(1000, requires_grad=True)
y = torch.randn(1000)
z = hist_loss(x, y)
z.backward()
x.grad is None
</code></pre>
The size of the lstsq() solution tensor is different from Normal equation

YeongHwa_Jin: <p>Let A is m x n tensor and B is m x k tensor.<br>
Then size of normal equation tensor is n x k.<br>
but the size of return tensor of torch.lstsq() is m x k.<br>
Why does it works like this?</p>

mlizhardy: <p>same question! and how do we get the proper n x k result?</p>
GPU Operations with constants

blackberry: <p>Hello…</p>
<p>Suppose I have a tensor <code>t</code> that resides on GPU and I would like to apply some math operations on it with constants:</p>
<pre><code class="lang-auto">t = torch.FloatTensor(...).cuda()
t = 0.1 * t + 0.2
</code></pre>
<p>I wonder how Torch/Python handles the constants <code>0.1</code> and <code>0.2</code>. Are they transferred from CPU to GPU memory each time this operation is performed, or is the code optimized automatically and the constants are moved to GPU?<br>
I am concerned about the efficiency.<br>
Should I define constant CUDA tensors and use them instead (less readable code)?</p>
<p>Thanks.</p>

googlebot: <p>it comes down to <em>two</em> cuda kernels that, strongly simplified, look like:<br>
mul(float* a, float* b)<br>
vs<br>
mul(float* a, float b)</p>
<p>second one is marginally faster, as one “load from global memory pointer” operation is avoided</p>
<p>but this is not related to CPU-GPU transfers, note that function arguments, launch configuration and a signal to execute cuda kernel itself require a data transfer anyway (I’d assume it is a single transfer that uses faster “constant memory”, but haven’t investigated into this).</p>
<p>takeaway is, it is better to NOT use tensors, when tensor-scalar operator versions exist (they usually do when you don’t need to wrap numbers in tensors). OTOH, the timing difference is negligible.</p>

blackberry: <p>Thanks for the nice explanation.</p>
<p>This is a very common use-case, and I almost always see constants being used as constants in the equations instead of constant tensors.<br>
There is also the caching; not familiar with GPU caching, but assuming it is similar to CPU caching, such constants are probably cached if they are frequently accessed.</p>

googlebot: <p>It is not about caching, what I tried to explain is that to call a kernel you either transfer a pointer to tensor data (address of 8 bytes) or a scalar (1-8 bytes), it is the same overhead, probably unavoidable (unless you would compile specialized kernels with less function arguments).</p>

blackberry: <p>I understand that, but the real cost comes from whether the value/pointer is accessed from cache or from GPU/CPU memory, the latter being much slower (especially from CPU memory).</p>

googlebot: <p>In that sense, I think special memory area is used to pass arguments, reading from which is at least not slower than from cached global memory. And I haven’t mentioned that the version with pointers requires doing address arithmetic in all threads, so it is worse than my oversimplified example suggests.</p>
I need help " TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first "

kelvin_sugi: <p>i trying to run this plot but is said error: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first , anyone know why?</p>
<pre><code class="lang-auto">def plot_training_history(history):
  fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(18, 6))

  ax1.plot(history['train_loss'], label='train loss')
  ax1.plot(history['val_loss'], label='validation loss')

  ax1.xaxis.set_major_locator(MaxNLocator(integer=True))
  ax1.set_ylim([-0.05, 1.05])
  ax1.legend()
  ax1.set_ylabel('Loss')
  ax1.set_xlabel('Epoch')

  ax2.plot(history['train_acc'], label='train accuracy')
  ax2.plot(history['val_acc'], label='validation accuracy')

  ax2.xaxis.set_major_locator(MaxNLocator(integer=True))
  ax2.set_ylim([-0.05, 1.05])
  ax2.legend()

  ax2.set_ylabel('Accuracy')
  ax2.set_xlabel('Epoch')

  fig.suptitle('Training history')
</code></pre>

rahulvigneswaran: <p>Some aspects of <code>matplotlib</code> will try to convert the input you provide it into <code>numpy</code>. It can do it only if the provided input tensor is in <code>cpu</code>.</p>
<p>So, to begin with I would suggest you to try and do <code>.cpu()</code> to the inputs that you provide, that is, check if every value in the dict <code>history</code> that you are using is on <code>cpu</code>.</p>
Resent-50 model downloading every time I load my trained model

pineapple: <p>Every time I load my model it’s pulling resent model from PyTorch. Is this a normal behavior or am I doing something different for this happen? Shouldn’t the model I downloaded and fine-tuned be enough for making inference.</p>
<p>Downloading: “<a href="https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth" rel="noopener nofollow ugc">https://download.pytorch.org/models/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth</a>” to /tmp/hub/checkpoints/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth</p>

Dwight_Foster: <p>This may be happening if pretrained_backbone=True and the model cannot find the official pretrained weights in the usual path. If you just want to use your own weights you can set pretrained_backbone to False and just load your weights manually like this</p>
<pre><code class="lang-auto">model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained_backbone=False)
model.load_state_dict(torch.load(PATH))
</code></pre>

pineapple: <p>torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained_backbone=False) still downloads model from pytorch smaller size then previous. Is there a way I can load it from my directory and not download it from pytorch</p>
Pytorch seems to have suddenly stopped using GPU even tgough it worked perfectly fine before an Ubuntu reboot

SpaceExp_NN: <p>I am running an RL training sessions using Pytorch and I have been able to run all my training sessions on my GPU for past month without any issues. For some unknown reason my computer rebooted today and ever since that, I am not able to run my training code on the GPU. I keep getting an error that says “RuntimeError: CUDA error: all CUDA-capable devices are busy or unavailable”. I found a similar post on this forum that suggested rebooting the computer but that hasn’t worked either. I even purged and reinstalled all my NVIDIA drivers and I am able to recognize them from the terminal. I will attached a few software related specifics to this message. Any advice on how to resolve this issue would be greatly appreciated!</p>
<p>$ nvcc --version<br>
Cuda compilation tools, release 9.1, V9.1.85</p>
<p>$ conda list<br>
cudatoolkit               10.1.243<br>
pytorch                   1.8.1<br>
torch                     1.7.1                    pypi_0</p>
<p>$ nvidia-smi<br>
NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2</p>
<p>$ torch.cuda.is_available()<br>
False</p>
<p>$ torch.cuda.device_count()<br>
0</p>

ptrblck: <p>This issue can be hit e.g. if your system automatically updated some drivers and failed in doing so.<br>
Based on your information it seems that you have a local CUDA9.1 toolkit installation as well as a new 460 driver. Did you update this driver manually or could it have been updated by the system?</p>

SpaceExp_NN: <p>In my conda environment, I updated the CUDA manually because I read elsewhere that CUDAtoolkit 10.1 would resolve the issue (it didn’t). The 460 driver was installed after I purged all the NVIDIA drivers and re-installed it. I am not sure which cudatoolkit I had before I changed it to 10.1 but I am certain that I after 460 in there even before I purged it.<br>
EDIT: I just checked my nvidia-smi and it says that I have CUDA version 11.2 but my nvcc --version says that I have “Cuda compilation tools, release 9.1, V9.1.85”. Is this nomal or could this be my source of error?</p>
How to implement custom recurrent model without iterating over time?

Christian_S: <p>Hello,</p>
<p>I want to implement the Kalman Filter in Pytorch. For that I have written a custom<code> KalmanCell(nn.Module)</code> that performs a Kalman update/predict step for one time step.<br>
I want to integrate the Cell in a module that takes as inputs a time series and outputs the filtered/noise-free time series.</p>
<p>My question is, how do I implement the module that calls KalmanCell without iterating over all time steps of the input time series?</p>
<p>For now I have a loop over the length of the time series (T).</p>
<pre><code class="lang-auto">for t in range(self.T):
      x_post, P_post, x_prior, P_prior = self._KalmanCell(x_prior, P_prior, input[:,t])
</code></pre>
<p>Is there a way to get rid of the loop? Can I inherit from RNN class? If so how would I write this?</p>
<p>For now I only saw examples with loop over time such as this <a href="https://github.com/pytorch/examples/tree/master/time_sequence_prediction" rel="noopener nofollow ugc">time series example from Pytorch</a>.</p>
<p>I would appreciate any suggestions. Thanks!</p>
Need COllaborators for robust deep rl framework using LibTorch

Adesola_Adedewe: <p>I recently began developing a c++ drl framework with libtorch with a simplified interface to enable researchers take advantage of multi-threading to acquire more batches and also c++ computation speed. I have built PPO and A2C to utilize this feature and i am blown away by the results and efficiency of resource use compared to multiprocessing. I need to work with other drl experts or software engineers to really bring this to life completely, as i believe drl research easily is limited by your accessible to large computing power.</p>
Alter model layers of VGG and rebuild module structure

rap1ide: <p>Hey everyone,</p>
<p>I want to optimize a vanilla VGG model after training and delete unnecessary feature maps of BatchNorm2d (BN) and Conv2d (C2D) layers.</p>
<p>To achieve that I first flatten all layers of the vanilla VGG, get all the BN and C2D layers, modify them and put those flattened layers back into a nn.Sequential() as my new model.</p>
<p>This fails because in vanilla VGG model the forward pass looks like:</p>
<pre><code class="lang-auto">    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        x = self.features(x)
        x = self.avgpool(x)
        x = torch.flatten(x, 1) # -&gt; flattening tensor does not happen when using nn.Sequential(), shape mismatch
        x = self.classifier(x)
        return x
</code></pre>
<p>My question now is:<br>
How can I maintain the model structure when altering the layers? My algorithm currently relies on two lists of all BN and all C2D layers. Is there something like a unique ID a layer carries inside the model structure so that I can just replace it with something like:</p>
<pre><code class="lang-auto">model.modules['uniqueID'] = altered_layer
</code></pre>
<p>If there are no uniqueIDs can u imagine a way to somehow build these myself (encoding module structure and decoding it)?</p>
<p>I could just make it work and write my own model with this specific forward pass, but I consider replacing the layers to be more elegant.</p>
<p>Help is appreciated!</p>
<p>Christian</p>

rap1ide: <p>Meanwhile I tinkered some code which works:</p>
<pre><code class="lang-auto">import functools
import torch
</code></pre>
<pre><code class="lang-auto">def get_module_layers_and_keychains(module):
    """ Returns all layers of a pytorch module and a keychain as identifier.
        e.g.
        ...
        (['features', '0', '5'], nn.ReLU())
        ...
        (['classifier', '0'], nn.BatchNorm2D())
        (['classifier', '1'], nn.Linear())
    """
    flattened = []

    o_dict = module._modules
    for key, child in o_dict.items():
        if not child._modules:
            structure = [([key], child)]
        else:
            structure = get_module_layers_and_structure(child)
            structure = [([key] + keylist, child) for keylist, child in structure]

        flattened.extend(structure)

    return flattened
</code></pre>
<pre><code class="lang-auto">def replace_module_layer(module, layer_keychain, new_layer):
    """Replaces a layer inside a pytorch module for a given keychain.
       Use get_module_layers_and_keychains() to retrieve valid keychains.
    """
    get_fn = lambda self, key: self._modules[key]
    l_root = functools.reduce(get_fn, layer_keychain[:-1], module)
    l_root._modules[layer_keychain[-1]] = new_layer
</code></pre>
<p>I am still interested in your thoughts and ideas!</p>
<p>Cheers.</p>
Layer by layer backpropagation to modify the chain rule

AliSh: <p>Hello,</p>
<p>In the backward pass of training a DNN, I need to compute the gradient layer by layer and modify the gradient before passing it to the previous layer.</p>
<p>It would be great, if you could help me please!</p>

albanD: <p>Hi,</p>
<p>If you want to work at the nn.Module level, you can use backward hooks so that you can run arbitrary code alongside the backward pass: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=backward_hook#torch.nn.Module.register_full_backward_hook" class="inline-onebox">Module — PyTorch 1.8.1 documentation</a></p>
T5ConditionalGeneration on squad dataset Validation Loss not decreasing

Shivam_Marathe: <p>I am trying to fine tune the model. and model is training / learning the QA tasks and I am getting more better answers. But validation loss is increasing</p>
<p>Here is my function for training</p>
<pre><code class="lang-auto">for e in range(3):
    train_loss = 0.0
    model.train()
    tbar = tqdm(batches)
    counter = 0
    losses = []
    for index, batch in enumerate(tbar):
        # Transfer Data to GPU if available
        optimizer.zero_grad()
        
        input_ids, attention_mask, labels = batch['input_ids'], batch['attention_mask'], batch['labels']
        if torch.cuda.is_available():
            input_ids, attention_mask, labels = input_ids.cuda(), attention_mask.cuda(), labels.cuda()
          
        # Clear the gradients
        # Forward Pass
        output = model(
            input_ids=input_ids, 
            attention_mask=attention_mask,
            labels=labels)

        loss, target = output.loss, output.logits
        # Calculate gradients 
        loss.backward()
        # Update Weights
        optimizer.step()
        
        
        # Calculate Loss
        train_loss += (loss.item() * 8)
        counter += 1
        losses.append(train_loss / (counter * 8))
        tbar.set_postfix({'loss': train_loss / (counter * 8)})
    valid_loss = 0.0
    model.eval()     # Optional when not using Model Specific layer
    vbar = tqdm(validloader)
    counter = 0
    with torch.no_grad():
        for index, batch in enumerate(vbar):
            # Transfer Data to GPU if available
            input_ids, attention_mask, labels = batch['input_ids'], batch['attention_mask'], batch['labels']
            if torch.cuda.is_available():
                input_ids, attention_mask, labels = input_ids.cuda(), attention_mask.cuda(), labels.cuda()

            output = model(
                input_ids=input_ids, 
                attention_mask=attention_mask,
                labels=labels)

            loss, target = output.loss, output.logits
            # Calculate gradients 
            # Calculate Loss
            valid_loss += loss.item() 
            losses.append(valid_loss)
            counter += 1 
            vbar.set_postfix({'loss': valid_loss / (counter)})
        
    #tk1.close()
    filename =   './saved_model' + str(e) + '.pt'
    torch.save(model.state_dict(), filename)
    print(f'Epoch {e+1} \t\t Training Loss: {train_loss / len(trainloader)} \t\t Validation Loss: {valid_loss / len(validloader)}')
    
    if min_valid_loss &gt; valid_loss:
        print(f'Validation Loss Decreased({min_valid_loss:.6f}---&gt;{valid_loss:.6f}) \t Saving The Model')
        min_valid_loss = valid_loss
          
        # Saving State Dict
        torch.save(model.state_dict(), './saved_model.pth')
</code></pre>
<p>can someone please see into it ?</p>
<p>Training loss is decreasing but validation loss is increasing.</p>
<p>Also training dataset = 65K examples and valid set = 27K examples</p>
Pytorch model to TensorRT failed

jinfagang: <p>[8] Assertion failed: ctx-&gt;network()-&gt;hasExplicitPrecision() &amp;&amp; “TensorRT only supports multi-input conv for explicit precision QAT networks!”</p>
<p>Does any one able to solve this problem?</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/onnx/onnx-tensorrt/issues/645" target="_blank" rel="noopener nofollow ugc">github.com/onnx/onnx-tensorrt</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/onnx/onnx-tensorrt/issues/645" target="_blank" rel="noopener nofollow ugc">[8] Assertion failed: ctx-&gt;network()-&gt;hasExplicitPrecision() &amp;&amp; "TensorRT only supports multi-input conv for explicit precision QAT networks!"</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-02-19" data-time="03:55:04" data-timezone="UTC">03:55AM - 19 Feb 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/jinfagang" target="_blank" rel="noopener nofollow ugc">
          <img alt="jinfagang" src="https://avatars.githubusercontent.com/u/21303438?v=4" class="onebox-avatar-inline" width="20" height="20">
          jinfagang
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Hi, I try to convert a model of onnx with normal data type float32 not QAT models. But it gives me...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
How to make this model exported convertable to trt engine?

jinfagang: <p>I know this not related about pytorch but possible about pytorch symblic to onnx, I have this model and I converted to onnx, but somehow the model can not converted to tensorrt using onnx2trt:</p>
<pre><code class="lang-auto">class MG(nn.Module):

    def __init__(self):
        super().__init__()
        # for test if torch.cat([bool, bool]) can convert

    def forward(self, x, b):
        preds = F.conv2d(x, b,
                             stride=1)
        return preds


torch_model = MG()
x = torch.randn([1, 4, 24, 24])
b = torch.randn([8, 4, 3, 3])
torch_out = torch_model(x, b)

# Export the model
torch.onnx.export(torch_model,               # model being run
                  (x, b),
                  "a.onnx",
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=11,          # the ONNX version to export the model to
                  do_constant_folding=True,
                  verbose=True)
print('Done!')
</code></pre>
<p>It’s very simple model. I want accerlerate it with tensorrt, but using onnx2trt convert onnx, I got error:</p>
<pre><code class="lang-auto">  [8] Assertion failed: ctx-&gt;network()-&gt;hasExplicitPrecision() &amp;&amp; "TensorRT only supports multi-input conv for explicit precision QAT networks!"
</code></pre>
<p>A reference is <a href="https://github.com/onnx/onnx-tensorrt/issues/645" class="inline-onebox" rel="noopener nofollow ugc">[8] Assertion failed: ctx-&gt;network()-&gt;hasExplicitPrecision() &amp;&amp; "TensorRT only supports multi-input conv for explicit precision QAT networks!" · Issue #645 · onnx/onnx-tensorrt · GitHub</a></p>
<p>I found the onnx-tensorrt is no one maintain it but I really need it, so ask to pytorch community doesn anyone able to solve it?</p>
<p>BTW: this model seems can converted in pytorch 1.6 but can not converted to pytorch 1.7.</p>

jinfagang: <p>Does anyone help me out?</p>

jinfagang: <p><a class="mention" href="/u/ptoews">@ptoews</a>  Can u help me?</p>
Can I get gradients of network for each sample in the batch?

Olzhas_Kabdolov: <p>Hi! I am implementing a simple algorithm to compute squared variance of gradients. For this, I need to compute gradients of the network for each sample in my batch. As far as I understand, loss.backward() computes gradients for each sample and then averages them. Is there any flag which outputs gradients for each sample before their averaging? Thanks!</p>

JuanFMontesinos: <p>As far as I know you cannot get specific grads unless you backpropagate for each loss term before averaging.<br>
You can get element-wise loses by passing reduce=False<br>
Check any loss’ docs to see<br>
<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.L1Loss" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#torch.nn.L1Loss</a></p>
<p>So, in short,</p>
<pre><code class="lang-auto">criterion = nn.Loss(reduce=False)

...
losses = criterion(pred,gt)
for loss in losses:
    loss.backward(retain_graph=True)
</code></pre>

Olzhas_Kabdolov: <p>Hi <a class="mention" href="/u/juanfmontesinos">@JuanFMontesinos</a> ! Thank for reply! Yes, I am aware of loss “reduction”. However, I need gradients of the network itself. At the moment, I am forwarding and backwarding each sample, which is very slow.</p>

JuanFMontesinos: <p>As I aforementioned, you can backpropagate each element before reduction. At least this will speed up forward.<br>
For backward, it has to be 1-by-1.<br>
I think the whole system is built under the idea that you can access gradients doing param.grad, that’s why it would be a mess if you could filter out gradients from any element. Besides, the system is not aware of what “loss”  or a batch is is. It just follows chain rule.</p>

dejanbatanjac: <p>Just a minor, <code>reduce</code> h/b deprecated. Use <code>reduction</code> instead:</p>
<blockquote>
<ul>
<li><strong>reduction</strong> ( <em>string</em> <em>,</em> <em>optional</em> ) – Specifies the reduction to apply to the output: <code>'none'</code> | <code>'mean'</code> | <code>'sum'</code> . <code>'none'</code> : no reduction will be applied, <code>'mean'</code> : the sum of the output will be divided by the number of elements in the output, <code>'sum'</code> : the output will be summed. Note: <code>size_average</code> and <code>reduce</code> are in the process of being deprecated, and in the meantime, specifying either of those two args will override <code>reduction</code> . Default: <code>'mean'</code></li>
</ul>
</blockquote>
<p>Probable you are looking for option: <code>none</code></p>

albanD: <p>If you use simple  NN, you can use tricks like the one mentionned <a href="https://github.com/cybertronai/autograd-hacks">here</a> to reuse computations.</p>

AliSh: <p>Hi <a class="mention" href="/u/alband">@albanD</a><br>
I have a customised layer as will in my NN. So is there anyway to still use the trick that you mentioned please?</p>
Customizing Tensorboard's Add Embedding

ecdrid: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> ! (Apologies for @)</p>
<p>I was just wondering, How can you customize the Tensor-board’s Summary Writer that we have in PyTorch  to something as shown in the below picture?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/414433f4a8633ca682d9b99ef6d64620bf4f1d04.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/414433f4a8633ca682d9b99ef6d64620bf4f1d04" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/414433f4a8633ca682d9b99ef6d64620bf4f1d04_2_518x499.jpeg" alt="image" data-base62-sha1="9jn9U04X6XTBYMTTyutp6VfgHNG" width="518" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/414433f4a8633ca682d9b99ef6d64620bf4f1d04_2_518x499.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/4/1/414433f4a8633ca682d9b99ef6d64620bf4f1d04.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/1/414433f4a8633ca682d9b99ef6d64620bf4f1d04.jpeg 2x" data-dominant-color="EBE1DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">587×566 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks for your response,<br>
Aditya.</p>
Create my own dataset from tensor

DanielC: <p>Hi there,<br>
I have a torch tensor whose size is [100000, 15, 2] and I want to use it as my dataset (because I am working with GANs so no label needed). and here is my code:</p>
<pre><code class="lang-python">shuffle = True
batch_size = 125
num_worker = 2
pin_memory = True

tensor_input_data = torch.Tensor(input_data) 

my_dataset = TensorDataset(tensor_input_data)
my_dataloader = DataLoader(dataset= my_dataset, 
                            shuffle=shuffle,
                            batch_size=batch_size,
                            num_workers=num_worker,
                            pin_memory=pin_memory)

</code></pre>
<p>when I load the dataloader for training</p>
<pre><code class="lang-auto">for epoch in range(n_epochs):
        for real in tqdm(dataloader):
            cur_batch_size = len(real)
            real = real.view(cur_batch_size, -1).to(device)
</code></pre>
<p>It has the traceback that says ‘list’ object has no attribute ‘view’, I know this mean that “real” is a list, but why?</p>
<p>Appreciate any suggestions！</p>

pascal_notsawo: <p>You can replace the line <code>real = real.view(cur_batch_size, -1)</code> with <code>real = torch.stack(real).view(cur_batch_size, -1)</code>, or <code>real = torch.stack(real).view(-1)</code> (depending on the dimensions of your data, you do not need to specify <code>cur_batch_size</code>)</p>

KyleOng: <p><code>TensorDataset</code> returns a tuple when the dataloader calls <code>__getitem__</code>.</p>
<p>See source code here: <a href="https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#TensorDataset" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data.dataset — PyTorch 1.8.1 documentation</a></p>
UnavailableInvalidChannel: The channel is not accessible or is invalid

helloybz: <p>Hi, I’m trying to install pytorch 1.8.1 from conda by:</p>
<pre><code class="lang-bash">conda install -c pytorch pytorch
</code></pre>
<p>and it results:</p>
<pre><code class="lang-bash">Collecting package metadata (current_repodata.json): failed

UnavailableInvalidChannel: The channel is not accessible or is invalid.
  channel name: pytorch
  channel url: https://conda.anaconda.org/pytorch
  error code: 404

You will need to adjust your conda configuration to proceed.
Use `conda config --show channels` to view your configuration's current state,
and use `conda config --show-sources` to view config file locations.
</code></pre>
<p>conda config --show channels results:</p>
<pre><code class="lang-bash">channels:
  - defaults
</code></pre>
<p>conda config --show-sources results nothing</p>
<hr>
<p>torchaudio is also not installed, but another packages, like librosa, is installed well.<br>
What can I do?</p>
<p>macOS, miniconda 4.10.1.</p>

ptrblck: <p>Are you able to open the channel URL and are you seeing all stored packages or does the URL also yield a 404?<br>
I can install the current binaries (did it a couple of times just a few minutes ago) and can also access the URL.</p>

helloybz: <p>Yes, i can access to the url via web browsers and can see the packages i want to install.</p>
<p>I’ve just tried to install same things, but on aws ec2 instances (ubuntu18.04, with gpu), all the things works well.</p>

helloybz: <p>I’ve found it is cause from the conda side.</p>
<p>I thought there were changes in installing pytorch in macOS. thank you.</p>
Loss value is kept constant in the training

Davide_Murari: <p>I am new to Pytorch and I am implementing a simple feedforward neural network and the loss function does not seem to decrease.</p>
<p>Because of some other tests I have done, the problem seems to be in the computations I do to compute <em>pred</em>, since if I slightly change the network so that it spits out a 2-dimensional vector for each entry and save it as pred, everything works perfectly.</p>
<p>Do you see the problem in defining pred here? Thanks</p>
<p>As far as I understand, what is happening is that XH forgets the dependency on X, and hence even params does so.</p>
<pre><code>import torch
import numpy as np
from torch import nn

dt = 0.1

class Neural_Network(nn.Module):
    def __init__(self, ):
        super(Neural_Network, self).__init__()
    
        self.l1 = nn.Linear(2,300)
        self.nl = nn.Tanh()
        self.l2 = nn.Linear(300,1)
    
    
    def forward(self, X):
        z = self.l1(X)
        z = self.nl(z)
        o = self.l2(z)
        return o



N = 1000
X = torch.rand(N,2,requires_grad=True)
y = torch.rand(N,1)
NN = Neural_Network()

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(NN.parameters(), lr=1e-5)

epochs = 200

for i in range(epochs):  # trains the NN 1,000 times

    HH = torch.mean(NN(X))
    gradH =  torch.autograd.grad(HH, X)[0]
    XH= torch.cat((gradH[:,1].unsqueeze(0),-gradH[:,0].unsqueeze(0)),dim=0).t()
    pred = X + dt*XH

    #Optimize and improve the weights
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


    print (" Loss: ", loss.detach().numpy())  # mean sum squared loss
</code></pre>
<p>P.S. With these X and y the loss is not expected to go to zero, I have added them here like them just for simplicity. I will apply this architecture to data points which are expected to satisfy this model. However I am just interested in seeing the loss decreasing.</p>
Unexpected key(s) in state dict

jS5t3r: <p>I trained a model like this:</p>
<pre><code class="lang-auto">net = resnet50(num_classes=settings.NUM_CLASSES)
net = nn.parallel.DataParallel(net, device_ids=list(range(torch.cuda.device_count())))
net = net.cuda()
...
torch.save(net.state_dict(), weights_path)
</code></pre>
<p>I would like to load the model:</p>
<pre><code class="lang-auto">model = get_network(args)
model.load_state_dict(torch.load(settings.CHECKPOINT_PATH))
model = nn.parallel.DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.cuda()
model.eval()
</code></pre>
<p>I get this error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "attack.py", line 53, in &lt;module&gt;
    model.load_state_dict(torch.load(settings.CHECKPOINT_PATH))
  File "/home/user/.conda/envs/bm/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1052, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for ResNet:
        Missing key(s) in state_dict: "conv1.0.weight", "conv1.1.weight", "conv1.1.bias", "conv1.1.running_mean", "conv1.1.running_var", "conv2_x.0.residual_function.0.weight", "conv2_x.0.residual_function.1.weight", "conv2_x.0.residual_function.1.bias", "conv2_x.0.residual_function.1.running_mean", "conv2_x.0.residu
</code></pre>

ptrblck: <p>You are saving the <code>nn.DataParallel</code> <code>state_dict</code>, which will include the <code>.module</code> keys as described <a href="https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models">here</a>. You could either wrap the model first into <code>nn.DataParallel</code> and load the <code>state_dict</code> afterwards or store the <code>net.module.state_dict()</code> instead.</p>
Inherit a child class from ImageFolder

Deeply: <p>I am trying to customize <code>ImageFolder</code> to return the image, category and sub-category names. Yet, here’s what worked for my case:</p>
<pre><code class="lang-auto"> def __getitem__(self, index):        
        return super(ImageFolder, self).__getitem__(index), self.imgs[index]
</code></pre>
<p>Are there any better ways?</p>

ptrblck: <p>This looks like a quick and valid approach. You could of course derive a custom class from <code>ImageFolder</code>, but if you don’t need any other changes, I would probably just use yours.</p>

Deeply: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
It worked pretty well, as I passed a loader that read the images according to my needs, and a custom transform that plays with the images the way I want. Needless to say that <code>DatasetFolder</code> is also a very nice option/alternative as one can pass the extensions of the files to load.</p>
Using different feature maps for the backward pass than the forward pass

CaramelCougar: <p>I am doing a final project for a class and I need to do a proof of concept for an idea we have. Implementing the whole idea is out of scope for the course but something we need to do for the proof of concept is to be able to use different feature maps in the backward pass than the forward pass. Right now I only care about doing this for a Conv2D layer. I’ve messed with different stuff like trying to mess with the gradient graph (obviously didn’t work) and I think my best option is to define an autograd function. For the forward pass I know I can just use the functional API, but for the backward is there anything in pytorch that would allow me to easily get the gradient of a conv2d layer so i can easily compute the conv2d gradient in the backward function? Also how can i use an autograd function as a layer in a Sequential model? What would I have to wrap it in? The idea is that in the forward pass we compute the feature maps but throw them out and in the backward pass we need to recompute the feature maps to use for gradient calculations, but instead of recomputing them with the normal weights, we would recompute them with an approximated version of the weights.</p>

CaramelCougar: <p>nn.grad had the gradient functions I wanted. My only remaining question is how can I turn a custom autograd function into a layer I can input into a Sequential model?</p>

ptrblck: <p>You can define a custom <code>autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a> and use it in a custom <code>nn.Module</code> by falling it inside the <code>forward</code> method of the module.</p>
Pad Input HuggingFace

newbieee: <p>I want to pad my Inputs to Finetune a Wav2Vec Modell, but i cant.</p>
<pre><code class="lang-python">from transformers import Wav2Vec2Processor
model_name = 'maxidl/wav2vec2-large-xlsr-german'
processor = Wav2Vec2Processor.from_pretrained(model_name)

fake_batch = [{'input_values' : torch.rand([1, 112000])}, {'input_values' : torch.rand([1, 86000])}]
processor.pad(fake_batch, padding=True)
</code></pre>
<p>Is giving me</p>
<pre><code class="lang-auto">ValueError: Unable to create tensor, you should probably activate padding with 'padding=True' to have batched tensors with the same length.
</code></pre>
<p>Even tho i cleary set  ‘padding=True’. Whad am i doing wrong?</p>
<p>Ty in advanced</p>

ptrblck: <p>It seems that <a href="https://huggingface.co/transformers/model_doc/wav2vec2.html#transformers.Wav2Vec2FeatureExtractor.__call__"><code>Wav2Vec2FeatureExtractor.__call__</code></a> expects the <code>padding</code> argument, not the <code>pad</code> method, but I’m also not deeply familiar with all transformers calls, so you might want to post the question in the <a href="https://discuss.huggingface.co/">Huggingface forum</a>. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
How to use LayoutLM

dario: <p>Hi, I’m new to PyTorch. I wanted to implement a model for extracting structured data from forms or invoices. I came across this model <a href="https://github.com/microsoft/unilm/blob/master/layoutlm/layoutlm/modeling/layoutlm.py" rel="noopener nofollow ugc">LayoutLM</a>, which I believe should to the trick.</p>
<p>However, I’m kind of lost on where to start. Is there any documentation or examples out there on how to use the model?</p>
<p>I also found <a href="https://1drv.ms/u/s!ApPZx_TWwibInS3JD3sZlPpQVZ2b?e=bbTfmM" rel="noopener nofollow ugc">this on OneDrive</a>. Could I use this to simply import the model?</p>

ptrblck: <p>The <a href="https://github.com/microsoft/unilm/tree/master/layoutlm">repository</a> seems to contain some examples as well as install instructions, which might be helpful to get started.</p>