Question:
Hi there,
I am new to python and pytorch and stuck with my “little” project. I would like to train a model to predict a single value if it is provided a record of 30 values. I have a tensor of features and targets (provided below) for training purposes and a similar set for validation.
While writing this I stumbled upon the feature structure and I guess the feature tensor isnt correct. It should be a list of 8760 *3 records but it is a list of 3 containing a list of 8760 values. Is there an easy way to fix this (I ll have a look at the flatten function)?
It should select a batch of 10 records out of 8760*3 records for each training iteration.
I would like to get feedback on my code, the model a hint regarding the error and if the goal could be achieved with my model.
import torch
from torch import nn
from torch.utils.data import random_split, DataLoader, TensorDataset
import numpy as np
import ann_model as annM
import feature_selections as fs

# Get cpu or gpu device for ann_training.
device = "cuda" if torch.cuda.is_available() else "cpu"
print("Using {} device".format(device))

dataset_filename = 'file.npy'
dataset_filename = 'file2.npy' # sample input

print("Loading dataset: started")
print(dataset_filename, " will be loaded")
rawDataSet = np.load(dataset_filename, allow_pickle=True)
print("Loading dataset: finished")

# normierung aller Daten ?

print("Specify columns used for the moddeling dataset")
featureSelections = fs.feature_selection()
# https://stackoverflow.com/questions/44429199/how-to-load-a-list-of-numpy-arrays-to-pytorch-dataset-loader
featureData = torch.tensor(np.array(rawDataSet[:][featureSelections.s02()["names"]].tolist()))
targetsData = torch.tensor(np.array(rawDataSet[:]["trafo_q"].tolist()))

print("build pytorch dataset")
dataset = TensorDataset(featureData, targetsData)

print("split dataset into train and test")
train_size = int(0.8 * len(dataset))
test_size = len(dataset) - train_size
train, test = random_split(dataset, [train_size, test_size])

print("Create train- and testsets with dataloader and batchsze = 10")
batch_size = 10
trainset = DataLoader(train, batch_size=batch_size, shuffle=True)
testset = DataLoader(test, batch_size=batch_size, shuffle=True)

print("Creating NeuralNetwork Model")
numberOfFeatures = len(featureSelections.s02()["names"]) # number of features per record
model = annM.NeuralNetwork(numberOfFeatures) # create Model
model.to(device)

print("Optimizing Parameters")
optimizer = torch.optim.SGD(model.parameters(), lr = 0.9) # lr=1e-3
criterion = nn.BCELoss()

## TRAINING
print("Training NeuralNetwork")
model.train()
epochs = 5000
errors = []
for epoch in range(epochs):
    optimizer.zero_grad() # sets gradients to zero
    for feature, target in trainset:
        # Forward pass
        y_pred = model.forward(feature)
        # Compute Loss
        loss = criterion(y_pred.squeeze(), target)
        errors.append(loss.item())

        print('Epoch {}: train loss: {}'.format(epoch, loss.item()))
        # Backward pass
        loss.backward()
        optimizer.step()

### MODEL
import torch.nn as nn
import torch.nn.functional as F

# Define model
class NeuralNetwork (nn.Module):
    def __init__(self, numberOfFeatures):
        super(NeuralNetwork, self).__init__()
        self.l0 = nn.Linear(numberOfFeatures, 1024)
        self.l1 = nn.Linear(1024, 512)
        self.l2 = nn.Linear(512, 128)
        self.l3 = nn.Linear(numberOfFeatures, 1)

    def forward(self, inputs):
        output = self.l0(inputs)
        output = F.relu(self.l1(output))
        output = F.relu(self.l2(output))
        output = F.relu(self.l3(output))
        return output

    def training(self, trainset, optimizer, criterion):
        # self.model.train() # et mode to train other mode model.eval() called outside of class
        epochs = 5000
        errors = []
        for epoch in range(epochs):
            optimizer.zero_grad() # sets gradients to zero
            for feature, target in trainset:
                # Forward pass
                y_pred = self.forward(feature)
                # Compute Loss
                loss = criterion(y_pred.squeeze(), target)
                errors.append(loss.item())

                print('Epoch {}: train loss: {}'.format(epoch, loss.item()))
                # Backward pass
                loss.backward()
                optimizer.step()

## TRAINING
model.train()
epochs = 5000
errors = []
for epoch in range(epochs):
    optimizer.zero_grad() # sets gradients to zero
    for feature, target in trainset:
        # Forward pass
        y_pred = model.forward(feature)
        # Compute Loss
        loss = criterion(y_pred.squeeze(), target)
        errors.append(loss.item())

        print('Epoch {}: train loss: {}'.format(epoch, loss.item()))
        # Backward pass
        loss.backward()
        optimizer.step()

## FEATURES
>>> feature
tensor([[[ 1.0000e+00, -6.5955e+01,  1.4160e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 2.0000e+00, -6.4064e+01,  1.4160e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 3.0000e+00, -6.1796e+01,  1.6460e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         ...,
         [ 8.7580e+03, -1.0670e+02,  4.3340e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 8.7590e+03, -1.0681e+02,  4.1720e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 8.7600e+03, -1.0487e+02,  4.3040e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00]],

        [[ 1.0000e+00, -7.6689e+01,  1.4160e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 2.0000e+00, -7.4489e+01,  1.4160e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 3.0000e+00, -7.1851e+01,  1.6460e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         ...,
         [ 8.7580e+03, -1.2383e+02,  4.3340e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 8.7590e+03, -1.2397e+02,  4.1720e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00],
         [ 8.7600e+03, -1.2170e+02,  4.3040e+01,  ...,  2.0000e+00,
           0.0000e+00,  2.0000e+00]],

        [[ 1.0000e+00, -3.7761e+01,  4.6055e+03,  ...,  2.7200e+02,
           1.6400e+02,  2.8600e+02],
         [ 2.0000e+00, -5.0640e+01,  4.6055e+03,  ...,  2.7200e+02,
           1.6400e+02,  2.8600e+02],
         [ 3.0000e+00, -4.5840e+01,  4.9473e+03,  ...,  2.7200e+02,
           1.6400e+02,  2.8600e+02],
         ...,
         [ 8.7580e+03,  3.6286e+01,  2.6821e+04,  ...,  2.7200e+02,
           1.6400e+02,  2.8600e+02],
         [ 8.7590e+03,  4.0603e+01,  2.6394e+04,  ...,  2.7200e+02,
           1.6400e+02,  2.8600e+02],
         [ 8.7600e+03,  3.9810e+01,  2.6441e+04,  ...,  2.7200e+02,
           1.6400e+02,  2.8600e+02]]], dtype=torch.float64)
>>> feature.shape
torch.Size([3, 8760, 30])

## TARGETS

>>> target
tensor([[ 31.6668,  33.5949,  35.8350,  ..., -26.1178, -24.8364, -24.1894],
        [ 36.8206,  39.0619,  41.6658,  ..., -30.3122, -28.8252, -28.0726],
        [  3.4025,   6.6635,   3.2758,  ..., -12.6210, -15.8263, -15.4595]],
       dtype=torch.float64)
torch.Size([3, 8760])

## ERROR
>>> y_pred = model.forward(feature)
Traceback (most recent call last):
  File "~Eclipse-2020-09\dropins\PyDev 8.0.1\plugins\org.python.pydev.core_8.0.1.202011071328\pysrc\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<console>", line 1, in <module>
  File "~workspace\q_estimation\ann_model.py", line 24, in forward
    output = self.l0(inputs)
  File "~Python\Python38\site-packages\torch\nn\modules\module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "~Python\Python38\site-packages\torch\nn\modules\linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "~\Python\Python38\site-packages\torch\nn\functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: expected scalar type Double but found Float

Context:
You are feeding float64 tensors to the model, which uses float32 parameters so you would have to transform the inputs via x = x.float() before feeding them to the model.
Also, your last non-linearity is F.relu while nn.BCELoss expects probabilities in [0, 1]. I would recommend to remove F.relu and pass the raw logits to nn.BCEWithLogitsLoss instead for a better stability than sigmoid + nn.BCELoss.


Question:
return F.conv2d(input, weight, bias, self.stride,

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8, 4, 3, 3], but got 1-dimensional input of size [16] instead
class QNetwork(nn.Module):
def init(self, in_channels=4, num_actions=200):
super(QNetwork, self).init()
self.conv1 = nn.Conv2d(in_channels, 8, kernel_size=3, stride=1, padding=1)
self.conv2 = nn.Conv2d(8, 16, kernel_size=4, stride=2)
self.conv3 = nn.Conv2d(64, 64, kernel_size=3, stride=1)
self.fc4 = nn.Linear(7 * 7 * 64, 512)
self.fc5 = nn.Linear(512, num_actions)
def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.relu(self.conv2(x))
    x = F.relu(self.conv3(x))
    x = F.relu(self.fc4(x.view(x.size(0), -1)))
    return self.fc5(x)

def select_action(self, x):
    with torch.no_grad():
        fc5 = self.forward(x)
        action_index = torch.argmax(fc5, dim=1)
    return action_index.item()

Context:
Hi Anne!

 amahoro12:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [8, 4, 3, 3], but got 1-dimensional input of size [16] instead
…
self.conv1 = nn.Conv2d(in_channels, 8, kernel_size=3, stride=1, padding=1)

You are passing a 1D tensor of shape [16] into your model as its
input.  But the first layer of your model is a Conv2d (with
in_channels = 4).
Your Conv2d layer, and hence your model, requires an input of shape
[nBatch, in_channels = 4, height, width].  (nBatch can be 1,
but it has to be there.)
You would typically be passing into your model a batch of multi-channel
2D images.  (Because of your choice of having in_channels = 4 for
conv1, in this specific case you need to pass in a batch of four-channel
images.)
Best.
K. Frank


What’s the right way of implementing policy gradient?

Question:
Hello everyone!
I’m trying to perform this gradient update directly, without computing loss.

But I simply haven’t seen any ways I can achieve this. Looks like first I need some function to compute the gradient of policy, and then somehow feed it to the backward function.

Context:
I think you should look at this topic explaining the tensor.reinforce method:

    What is action.reinforce(r) doing actually? reinforcement-learning

    Hi,
I am studying RL with reinforcement/reinforce.py in pytorch/examples. I have some questions about it.

What does action.reinforce® internally do ?

Below is REINFORCE update rule where v_t is a return. We need to do gradient “ascent” as below but if we use optimizer.step, it is gradient “descent”. Is [action.reinforce®]((https://github.com/pytorch/examples/blob/master/reinforcement_learning/reinforce.py) multiplying log probability by -r? Then, it makes sense.
[image]

In autograd.ba…

If the action is the result of a sampling, calling action.reinforce(r) acts as a policy gradient.
You can find a code example of implementation here:

      github.com

    pytorch/examples/blob/master/reinforcement_learning/reinforce.py 1.5k
import argparse
import gym
import numpy as np
from itertools import count

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.distributions import Categorical

parser = argparse.ArgumentParser(description='PyTorch REINFORCE example')
parser.add_argument('--gamma', type=float, default=0.99, metavar='G',
                    help='discount factor (default: 0.99)')
parser.add_argument('--seed', type=int, default=543, metavar='N',
                    help='random seed (default: 543)')
parser.add_argument('--render', action='store_true',
                    help='render the environment')
parser.add_argument('--log-interval', type=int, default=10, metavar='N',

  This file has been truncated. show original


Question:
I have the following dqn model:
class DQN(nn.Module):

    def __init__(self, input_shape, n_actions):
        super(DQN, self).__init__()

        self.conv = nn.Sequential(
            nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
            nn.ReLU(),
            nn.Conv2d(32, 64, kernel_size=4, stride=2),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1),
            nn.ReLU()
        )

        conv_out_size = self._get_conv_out(input_shape)
        self.fc = nn.Sequential(
            nn.Linear(conv_out_size, 512),
            nn.ReLU(),
            nn.Linear(512, n_actions)
        )

    def _get_conv_out(self, shape):
        o = self.conv(Variable(torch.zeros(1, *shape)))
        return int(np.prod(o.size()))

    def forward(self, x):
        fx = x.float() / 256.0
        conv_out = self.conv(fx).view(fx.size()[0], -1)
        return self.fc(conv_out)

I trained it for Pong and got more than 18.0 mean reward for the last 100 games.
It was saved:
torch.save(net.state_dict(), 'pong_model.pt')

But when I try to load it and play some games I get wrong results.
net = models.DQN(env.observation_space.shape, env.action_space.n)
net.load_state_dict(torch.load('pong_model.pt', map_location=lambda storage, loc: storage))
for i in range(20):
    state = env.reset()
    while True:
        env.render()
        action = agent(state, net)
        next_state, _, done, _ = env.step(action)
        if done:
            break
        state = next_state

It can’t win even once. What could be wrong?

Context: Can you post the code for ‘agent()’?


Activation function and distribution for action space between 0 and 1?

Question:
Hi,
I am building my first REINFORCE (policy gradient) model with a continuous action space between 0 and 1. Right now, I use the following code for this:
        self.loc_layer = nn.Sequential(
            nn.Linear(size, n_params),
            nn.Sigmoid()
        )
        self.scale_layer = nn.Sequential(
            nn.Linear(size, n_params),
            nn.Sigmoid()
        )

    	# ...

        loc = self.loc_layer(x)
        if self.training:
            scale = self.scale_layer(x)
            dist = Normal(loc=loc, scale=scale)
            params = dist.sample()
            log_prob = dist.log_prob(params).sum(dim=1)
            params = params.clamp(min=0, max=1)
        else:
            params = loc
            log_prob = None

However, this does not seem right to me. For example, I do not like that I have to clamp to ensure that the actions (params) are between 0 and 1. Therefore, I consider using a Beta or Logit-Normal distribution instead. In other examples, I have seen that Softplus is often used as the activation function for the scale argument. But this does not seem right here because I want to limit the action space.
Can anyone with some experience in this area give me a recommendation to which activation function and distribution to use in this case?

Context:
Three solutions:

use a normal distribution, use tanh as mu activation (to keep the center in range, prevent shifting too much) and then clamp, but you should do clamping only on the action sent to the environment, and not actions stored in buffers. In this way, you are not changing the pdf of your action, but changing the reward distribution.

use a normal distribution, use tanh to clamp your output right after sampling, but in order to calculate the log probability of your sample, you need to do this step, which is used in the Soft Actor Critic algorithm 16.

It should be relatively easy to deduce this equation

use a different distribution, but normally, the normal distribution performs the best in all algorithms, I have not tried Beta or Logit-Normal distribution and don’t know their performance. This method is theoretically possible, but result is not garanteed.


Implementation of Meta-Learning for Reinforcement Learning

Question:
I’m trying to implement the following algorithm ([1710.03463] Learning to Generalize: Meta-Learning for Domain Generalization 1)

The best approach I was able to come up with was the following pseudocode:
STEP:
	# 1. Copy the train model
	model_meta_test.load_state_dict(model_train.state_dict())
	optimizer_meta_test.load_state_dict(optimizer_train.state_dict())

	# Caluclate loss on the meta test model (A)
	loss_meta_test = self.get_loss(model_meta_test)

	# Optimize the meta test model
	self.optimize_network(loss_meta_test, model_meta_test, optimizer_meta_test)

	# Now I'm at ϴ' which means I can calculate the loss of my META TEST
	loss_meta_test = self.get_loss(model_meta_test)

	# Now I need to recalculate the loss on the Train Model (B).
	# Note that the loss at (A) and (B) should be the same.
	loss_train = self.get_loss(model_train)

	# Now I calculate the sum of both losses.
	tot_loss_train = loss_train + self.BETA * loss_meta_test.detach()

	# Finally Optimize the Train Model
	self.optimize_network(tot_loss_train, model_train, optimizer_train)

I have a few doubts.

Is the implementation is correct.
What would be a more efficient way of doing it? This is an overkill of calculations as I have to backup models and calculate the same loss twice for different models. With my limited knowledge, this was the best I could come up with. If anyone can enlighten me, I would greatly appreciate it.

Context: Solved this by looking at an example. GitHub - Pulkit-Khandelwal/mldg


Combining two pretrained models for Training

Question:
I am trying to use resnet18 and densenet121 as the pretrained models and have added 1 FC layer at the end of each network two change dimensions to 512 of both network and then have concatenated the outputs and passed to two final FC layers.
this can be seen here:
class classifier(nn.Module):
  def __init__(self,num_classes):
    super(classifier,self).__init__()

    self.resnet=models.resnet18(pretrained=True)

    self.rfc1=nn.Linear(512,512)

    self.densenet=models.densenet121(pretrained=True)
    self.dfc1=nn.Linear(1024,512)

    self.final_fc1=nn.Linear(1024,512)
    self.final_fc2=nn.Linear(512,num_classes)
    self.dropout=nn.Dropout(0.2)

  def forward(self,x):
    y=x.detach().clone()

    x=self.resnet.conv1(x)
    x=self.resnet.bn1(x)
    x=self.resnet.relu(x)
    x=self.resnet.maxpool(x)
    x=self.resnet.layer1(x)
    x=self.resnet.layer2(x)
    x=self.resnet.layer3(x)
    x=self.resnet.layer4(x)
    x=self.resnet.avgpool(x)
    x=x.view(x.size(0),-1)
    x=nn.functional.relu(self.rfc1(x))

    y=self.densenet.features(y)
    y=y.view(y.size(0),-1)
    y=nn.functional.relu(self.dfc1(y))

    x=torch.cat((x,y),0)
    x=nn.functional.relu(self.final_fc1(x))
    x=self.dropout(x)
    x=self.final_fc2(x)

    return x

Error:
size mismatch, m1: [1048576 x 1], m2: [1024 x 512] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283

but as i could see the densenet outputs 1024 feature to the last layer
Questions:

Is the implementation correct ?
Can this implementation work as a Ensemble

Context:
Your current implementation misses the relu as well as adaptive pooling layer from the DenseNet implementation as used here 5, which would create a tensor of [batch_size, 1024, 7, 7] for the output of self.densenet.features.
Also, you might want to concatenate the activation tensors in dim1 (feature dimension).
These changes should work:
...
    y = self.densenet.features(y)
    y = F.relu(y)
    y = F.adaptive_avg_pool2d(y, (1, 1))
    y = y.view(y.size(0), -1)
    # y = F.relu(self.dfc1(y)) remove this here, if you are using the first relu

    x = torch.cat((x,y), 1)
...


Question:
Running following returns false:
import torch
torch.cuda.is_available()
nvidia-smi output: (driver version seems compatible with CUDA version)
±----------------------------------------------------------------------------+
| NVIDIA-SMI 442.19       Driver Version: 442.19       CUDA Version: 10.2     |
|-------------------------------±---------------------±---------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1650   WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   49C    P8     5W /  N/A |    132MiB /  4096MiB |      0%      Default |
±------------------------------±---------------------±---------------------+
Using Python 3.7.6
Windows 10 laptop with NVIDIA GeForce  GTX 1650
Torch version 1.4.0
Running deviceQuery and bandwidthTest returns all tests PASS successfull
torch.version.cuda returns none.
torch.backends.cudnn.enabled returns true.
Could not find a solution, tried re-installing Conda, CUDA, drivers, Pytorch - did not help.
Solution found: conda remove cpuonly

Context: Please uninstall cpuonly in your conda environment. If torch.version.cuda returns none, then it means that you are using a CPU only binary.


AssertionError: Torch not compiled with CUDA enabled 11.4

Question:
Hello, I’m trying to run styleGAN3 and keep facing this problem
AssertionError: Torch not compiled with CUDA enabled
How to fix it?
ray@gamepc:~/stylegan3-main$ conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

ray@gamepc:~/stylegan3-main$  conda init stylegan3

(stylegan3) ray@gamepc:~/stylegan3-main$ python3 gen_images.py --outdir=out --trunc=1 --seeds=2 --network=https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-afhqv2-512x512.pkl
Loading networks from "https://api.ngc.nvidia.com/v2/models/nvidia/research/stylegan3/versions/1/files/stylegan3-r-afhqv2-512x512.pkl"...
No CUDA runtime is found, using CUDA_HOME='/usr'
Traceback (most recent call last):
  File "/home/ray/stylegan3-main/gen_images.py", line 143, in <module>
    generate_images() # pylint: disable=no-value-for-parameter
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/ray/stylegan3-main/gen_images.py", line 108, in generate_images
    G = legacy.load_network_pkl(f)['G_ema'].to(device) # type: ignore
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 852, in to
    return self._apply(convert)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 530, in _apply
    module._apply(fn)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 552, in _apply
    param_applied = fn(param)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 850, in convert
    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
  File "/home/ray/anaconda3/envs/stylegan3/lib/python3.9/site-packages/torch/cuda/__init__.py", line 166, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

(stylegan3) ray@gamepc:~/stylegan3-main$ nvidia-smi
Wed Jan 19 22:51:34 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.86       Driver Version: 470.86       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:01:00.0 Off |                  N/A |
|  0%   60C    P2   385W / 420W |   9419MiB / 24251MiB |    100%      Default |
|                               |                      |                  N/A |
-------------------------------+----------------------+----------------------+

(stylegan3) ray@gamepc:~/stylegan3-main$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

Context: Check the CUDA runtime used in your PyTorch install via torch.version.cuda and torch.cuda.is_available(), as you might have installed the CPU-only version.


Efficient way to create block diagonal with different block size

Question:
I want to create a mask matrix through torch.block_diag() method. However, each diagonal block has a different shape.
torch.block_diag(torch.ones((2,2)), torch.ones((3,3)), torch.ones((2,2)))

Please can I know if there is an efficient way to achieve this?

Context: Your code works fine so I’m unsure I understand the issue correctly. Are you seeing any errors?


Question:
Initial situation: I have written an API to serve my object detection models. The model is loaded onto the GPU. Subsequently, the inferences are executed. However, if I have a break in between (let’s say 5 minutes), the first inference after the break then takes significantly longer.
My question: Are there certain parameters to prevent this? What is happening with the model on the GPU?

Context:
Your GPU might go into an idle state and the wake up might take some time. You could enable the persistence demon as described in the docs 2:

This approach would prevent the kernel module from fully unloading software and hardware state when no user software was using the GPU.


Pytorch not working for me but same setting works for other user

Question:
I can not run a simple code as this
import torch
toTest=torch.rand(2,4)
toTest=toTest.to(“cuda”)
print(toTest)
it just stuck and nothing happens.
Note: My real code is something else, but that one is also doing same, runs fine on cpu but not on cuda.
Here are the possibilities I tried so far.
1- all other options go to the same problem, but 10.2 has a different error as this is not compatible with RTX 3090.

but nothing worked. Even I asked my server admin to create a new account for me. he created a new account and I just installed the below packages. These are the total packages installed in my conda env.
(cant upload pictures as I am a new user)
My least requirement is Pytorch 1.9.0 as it is required by PyGCL.
I even asked my admin to create a new user for me and then I installed it there. but no… still the same problem. While my other labmate is using 1.9.0+cu111 or the same server and he has no problem.
Any guess what could be a possible problem?
here is the stackoverflow version of the question as there i can upload pictures.

      stackoverflow.com

  PyTorch with GPU is not working but working fine with CPU 1

  pytorch, anaconda, gpu

  asked by

    Adnan Ali

  on 01:45AM - 15 Dec 21 UTC

Context:
Since the same system seems to work for other users, I would guess it’s a setup issue on your account.
Try to run any CUDA application (e.g. build the CUDA samples and execute them on all devices), use a docker container and see if it would be working there, check dmesg for any Xid error codes.


ONNXRuntimeError: failed:Node (Gather_346) Op (Gather) [ShapeInferenceError] axis must be in [-r, r-1]

Question:
Hello all,
Recently,  I’m trying to deploy my model in a real environment. The model is a Fater-RCNN based object recognition model, as proposed by Anderson et al Bottom-up-attention. The model is implemented with Detectron.
The first try was with a web service (Flask plus Redis Queue), which works but with delays due to connection and transition issues. Therefore, an efficient solution was wished.
The second try is to export the PyTorch model and use it in the JAVA application, leveraged by DJL or DL4J. For that, I need to export the model as a ONNX graph at first. Here is the work I done as so far.
import onnxruntime as ort
import numpy as np
import torch
import onnx
import cv2

// Initialize model with checkpoint
model = BoAModel()
model.eval()

// Load image
cv_image = cv2.imread(image_path)
// Transformation, transform_gen () is used to resize the image
image, transforms = T.apply_transform_gens(transform_gen, cv_image)

// Model input
dataset_dict = [torch.as_tensor(image.transpose(2, 0, 1).astype("float32"), device="cpu"), torch.as_tensor([float(image_shape[0]) / float(h)], device="cpu")]

// Export onnx
model_onnx_path = "torch_model_boa_jit_torch1.5.1.onnx"

batch_size = 1
torch.onnx.export(model=model,  # model to be exported
                  args= data,  # model input
                  f=model_onnx_path,  # where to save the model
                  export_params=True,  # store the trained parameter weights inside the model file
                  opset_version=11,  # the ONNX version to export the model to
                  do_constant_folding=False,  # whether to execute constant folding for optimization
                  input_names=['input'],  # the model's input names
                  output_names=['output'],  # the model's output names
                  dynamic_axes={'input': {0: 'batch_size'},
                                'output':{0: 'batch_size'}},
                  verbose=True)

By exporting, there were a couple of warnings that have been asked and googled. It is not critical as so far in my case. After the exporting, I had  a sanity check, as shown as follows:
# Load the ONNX model
model = onnx.load(model_onnx_path)

# Check that the model is well formed
onnx.checker.check_model(model)

# Print a human readable representation of the graph
print(onnx.helper.printable_graph(model.graph))

So far so good. So I tried to load it in ONNX Runtime  with the following codes:
sess_options = ort.SessionOptions()
# Below is for optimizing performance
sess_options = ort.SessionOptions()
# sess_options.intra_op_num_threads = 24
# ...
ort_session = ort.InferenceSession(model_onnx_path, sess_options=sess_options)

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

# Compute ONNX Runtime output prediction (use the same data that model works well)
ort_inputs = {ort_session.get_inputs()[0].name: [to_numpy(torch.unsqueeze(torch.as_tensor(image.transpose(2, 0, 1).astype("float32"), device="cpu"), dim=0)), [torch.as_tensor(1.6)]]}
ort_outs = ort_session.run(None, ort_inputs)
print("ort_outs", ort_outs)

Then I got a ONNXRuntime Error at line (ort.InferenceSession(model_onnx_path,):
onnxruntime.capi.onnxruntime_pybind11_state.Fail: [ONNXRuntimeError] : 1 : FAIL : Load model from torch_model_boa_jit_torch1.5.1.onnx failed:Node (Gather_346) Op (Gather) [ShapeInferenceError] axis must be in [-r, r-1]

As the error shows, there was a problem at node Gather_346, it is shown in the following figure:
onnx1088×585 73.9 KB

I’m not quite sure whether there was an error at this step (e.g., Tensor data is empty in the figure). I check the verbose log of export. This Gather operation was given in box_regression (at line widths = boxes[:, 2] - boxes[:, 0] + OFFSET, where the box prediction will be calculated based on proposed regions and anchors, as shown as below:
  # torch.onnx.export log
  %964 : Float(22800) = onnx::Gather[axis=1](%962, %963) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:169:0
  %965 : Float(0) = onnx::Constant[value=[ CPUFloatType{0} ]]()
  %966 : Tensor = onnx::Constant[value={0}]()
  %967 : Float(22800) = onnx::Gather[axis=1](%965, %966) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:169:0
  %968 : Float(22800) = onnx::Sub(%964, %967) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:169:0

The codes where the errors (Gather_346)  might be triggered possibly  are as follows:
    def apply_deltas(self, deltas, boxes):
        """
        Apply transformation `deltas` (dx, dy, dw, dh) to `boxes`.

        Args:
            deltas (Tensor): transformation deltas of shape (N, k*4)
            boxes (Tensor): boxes to transform, of shape (N, 4)
        """
        assert torch.isfinite(deltas).all().item(), "Box regression deltas become infinite or NaN!"
        boxes = boxes.to(deltas.dtype)
        print("apply_deltas boxes.shape", boxes.shape)
        OFFSET = 1
        widths = boxes[:, 2] - boxes[:, 0] + OFFSET
        heights = boxes[:, 3] - boxes[:, 1] + OFFSET
        ctr_x = boxes[:, 0] + 0.5 * widths
        ctr_y = boxes[:, 1] + 0.5 * heights

        wx, wy, ww, wh = self.weights
        dx = deltas[:, 0::4] / wx
        dy = deltas[:, 1::4] / wy
        dw = deltas[:, 2::4] / ww
        dh = deltas[:, 3::4] / wh

        # clamping too large values into torch.exp()
        dw = torch.clamp(dw, max=self.scale_clamp)
        dh = torch.clamp(dh, max=self.scale_clamp)

        pred_ctr_x = dx * widths[:, None] + ctr_x[:, None]
        pred_ctr_y = dy * heights[:, None] + ctr_y[:, None]
        pred_w = torch.exp(dw) * widths[:, None]
        pred_h = torch.exp(dh) * heights[:, None]

        pred_boxes = torch.zeros_like(deltas)
        pred_boxes[:, 0::4] = pred_ctr_x - 0.5 * pred_w  # x1
        pred_boxes[:, 1::4] = pred_ctr_y - 0.5 * pred_h  # y1
        pred_boxes[:, 2::4] = pred_ctr_x + 0.5 * pred_w  # x2
        pred_boxes[:, 3::4] = pred_ctr_y + 0.5 * pred_h  # y2
        print("pred_boxes.shape", pred_boxes.shape)
        return pred_boxes

Here are the configurations of the python environment:
# python working environment  (1.5.1)
*  virtualenv         15.0.1
   python              3.6
   torch               1.5.1
   torchvision         0.6.1
   onnx                1.10.2
   onnxruntime         1.9.0
   ubuntu              16.04
   CUDA                10.2 (440.33.01, not used in test as so far)

In addition,  I’m not sure whether it is a torch version issue, so I had another try with the following environment:
# python working environment (1.10.0)
*  virtualenv         15.0.1
   python              3.6
   torch               1.10.0
   torchvision         0.11.1
   onnx                1.10.2
   onnxruntime         1.9.0
   ubuntu              16.04
   CUDA                10.2 (440.33.01, not used in test as so far)

The export procedure was simply crashed in this working environment with the following gdb debug logs.
/media/WorkSpace/Development_Repository/Working/models/detectron2/modeling/poolers.py:73: TracerWarning: Using len to get tensor shape might cause the trace to be incorrect. Recommended usage would be tensor.shape[0]. Passing a tensor of different shape might lead to errors or silently give incorrect results.
  (len(box_tensor), 1), batch_index, dtype=box_tensor.dtype, device=box_tensor.device

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007fffc9e57c08 in std::_Function_handler<void (onnx_torch::InferenceContext&), onnx_torch::OpSchema onnx_torch::GetOpSchema<onnx_torch::ConstantOfShape_Onnx_ver9>()::{lambda(onnx_torch::InferenceContext&)#1}>::_M_invoke(std::_Any_data const&, onnx_torch::InferenceContext&) ()
   from /media/WorkSpace/Development_Repository/Working/models/venv3_torch/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so

So I stay with the 1.5.1 environment.  Additionally, I tested this exported ONNX model in JAVA with DL4J and got the following (as same as python) errors:
2021-11-25 15:41:30.1671963 [I:onnxruntime:, inference_session.cc:230 onnxruntime::InferenceSession::ConstructorCommon::<lambda_dcdcfd37ad4a704b0bd7e98f885edfd8>::operator ()] Flush-to-zero and denormal-as-zero are off
2021-11-25 15:41:30.1672832 [I:onnxruntime:, inference_session.cc:237 onnxruntime::InferenceSession::ConstructorCommon] Creating and using per session threadpools since use_per_session_threads_ is true
WARNING: Since openmp is enabled in this build, this API cannot be used to configure intra op num threads. Please use the openmp environment variables to control the number of threads.
Exception caught java.lang.RuntimeException: Load model from C:\Users\ps\IdeaProjects\Demo_DL4J\src\main\resources\checkpoint_boa.onnx failed:Node (Gather_341) Op (Gather) [ShapeInferenceError] axis must be in [-r, r-1]

Question:
Could somebody help me to figure out what might be going on here? In this test, the model and data are launched on CPU. In order to narrow down the errors, I followed the steps as Exporting Fasterrcnn Resnet50 fpn to ONNX suggested  I works in my test environment 1.51. However, I do not make it successfully on my own model. To make sure the inference works correctly, I used the same data that the model usually works well.
Any input will be appreciated!

Context:
Was really confused me is: the model works well in the normal reference step. With the same data, there are some unexcepted hidden cases in the export.
In addition, I used the following trace function for getting a model saved by jit since torch.onnx.export and torch.jit.trace do the same work eventually. After calling the following function,
traced_model = torch.jit.trace(model, example_inputs=(data[0], data[1]))

I had the errors like
forward() takes 2 positional arguments but 3 were given

For solving this error, I modified the signature of forward function of my model. As so far it is quiet.  Go back to jit.trace.
I also changed the slice function  widths = boxes[:, 2] - boxes[:, 0] + OFFSET  to ==>
widths = torch.select(boxes, dim=1, index=2) - torch.select(boxes, dim=1, index=0) + OFFSET
to make sure that this step is correct.  However, I got the same error as the previous one,
RuntimeError: select() cannot be applied to a 0-dim tensor.

This means the input data boxes is empty, yet the tract log shows the data is not empty, as the following log ( of the apply_deltas function) shows, the widths have a size of 22800.
  %958 : Float(22800) = onnx::Gather[axis=1](%956, %957) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:169:0

The exported model has empty data for this operation, as shown in.
How could it happen at this step?
The rest complete logs are as follows:
  %959 : Float(0) = onnx::Constant[value=[ CPUFloatType{0} ]]()
  %960 : Tensor = onnx::Constant[value={0}]()
  %961 : Float(22800) = onnx::Gather[axis=1](%959, %960) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:169:0
  %962 : Float(22800) = onnx::Sub(%958, %961) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:169:0
  %963 : Float() = onnx::Constant[value={1}]()
  %964 : Float(22800) = onnx::Add(%962, %963) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:186:0
  %965 : Float(0) = onnx::Constant[value=[ CPUFloatType{0} ]]()
  %966 : Tensor = onnx::Constant[value={3}]()
  %967 : Float(22800) = onnx::Gather[axis=1](%965, %966) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:170:0
  %968 : Float(0) = onnx::Constant[value=[ CPUFloatType{0} ]]()
  %969 : Tensor = onnx::Constant[value={1}]()
  %970 : Float(22800) = onnx::Gather[axis=1](%968, %969) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:170:0
  %971 : Float(22800) = onnx::Sub(%967, %970) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:170:0
  %972 : Float() = onnx::Constant[value={1}]()
  %973 : Float(22800) = onnx::Add(%971, %972) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:187:0
  %974 : Float(0) = onnx::Constant[value=[ CPUFloatType{0} ]]()
  %975 : Tensor = onnx::Constant[value={0}]()
  %976 : Float(22800) = onnx::Gather[axis=1](%974, %975) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:171:0
  %977 : Float() = onnx::Constant[value={0.5}]()
  %978 : Float(22800) = onnx::Mul(%964, %977)
  %979 : Float(22800) = onnx::Add(%976, %978) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:184:0
  %980 : Float(0) = onnx::Constant[value=[ CPUFloatType{0} ]]()
  %981 : Tensor = onnx::Constant[value={1}]()
  %982 : Float(22800) = onnx::Gather[axis=1](%980, %981) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:172:0
  %983 : Float() = onnx::Constant[value={0.5}]()
  %984 : Float(22800) = onnx::Mul(%973, %983)
  %985 : Float(22800) = onnx::Add(%982, %984) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:185:0
  %986 : Tensor = onnx::Constant[value={1}]()
  %987 : Tensor = onnx::Constant[value={0}]()
  %988 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %989 : Tensor = onnx::Constant[value={4}]()
  %990 : Float(22800, 1) = onnx::Slice(%955, %987, %988, %986, %989) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:175:0
  %991 : Tensor = onnx::Constant[value={1}]()
  %992 : Tensor = onnx::Constant[value={1}]()
  %993 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %994 : Tensor = onnx::Constant[value={4}]()
  %995 : Float(22800, 1) = onnx::Slice(%955, %992, %993, %991, %994) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:176:0
  %996 : Tensor = onnx::Constant[value={1}]()
  %997 : Tensor = onnx::Constant[value={2}]()
  %998 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %999 : Tensor = onnx::Constant[value={4}]()
  %1000 : Float(22800, 1) = onnx::Slice(%955, %997, %998, %996, %999) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:177:0
  %1001 : Tensor = onnx::Constant[value={1}]()
  %1002 : Tensor = onnx::Constant[value={3}]()
  %1003 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1004 : Tensor = onnx::Constant[value={4}]()
  %1005 : Float(22800, 1) = onnx::Slice(%955, %1002, %1003, %1001, %1004) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:178:0
  %1006 : None = prim::Constant()
  %1007 : Double() = onnx::Constant[value={4.13517}]()
  %1008 : Tensor = onnx::Cast[to=1](%1007)
  %1009 : Float(22800, 1) = onnx::Clip(%1000, %1006, %1008) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:181:0
  %1010 : None = prim::Constant()
  %1011 : Double() = onnx::Constant[value={4.13517}]()
  %1012 : Tensor = onnx::Cast[to=1](%1011)
  %1013 : Float(22800, 1) = onnx::Clip(%1005, %1010, %1012) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:182:0
  %1014 : Float(22800, 1) = onnx::Unsqueeze[axes=[1]](%964) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:184:0
  %1015 : Float(22800, 1) = onnx::Mul(%990, %1014) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:184:0
  %1016 : Float(22800, 1) = onnx::Unsqueeze[axes=[1]](%979) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:184:0
  %1017 : Float(22800, 1) = onnx::Add(%1015, %1016) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:184:0
  %1018 : Float(22800, 1) = onnx::Unsqueeze[axes=[1]](%973) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:185:0
  %1019 : Float(22800, 1) = onnx::Mul(%995, %1018) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:185:0
  %1020 : Float(22800, 1) = onnx::Unsqueeze[axes=[1]](%985) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:185:0
  %1021 : Float(22800, 1) = onnx::Add(%1019, %1020) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:185:0
  %1022 : Float(22800, 1) = onnx::Exp(%1009) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:186:0
  %1023 : Float(22800, 1) = onnx::Unsqueeze[axes=[1]](%964) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:186:0
  %1024 : Float(22800, 1) = onnx::Mul(%1022, %1023) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:186:0
  %1025 : Float(22800, 1) = onnx::Exp(%1013) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:187:0
  %1026 : Float(22800, 1) = onnx::Unsqueeze[axes=[1]](%973) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:187:0
  %1027 : Float(22800, 1) = onnx::Mul(%1025, %1026) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:187:0
  %1028 : Tensor = onnx::Shape(%955)
  %1029 : Float(22800, 4) = onnx::ConstantOfShape[value={0}](%1028) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:189:0
  %1030 : Float() = onnx::Constant[value={0.5}]()
  %1031 : Float(22800, 1) = onnx::Mul(%1024, %1030)
  %1032 : Float(22800, 1) = onnx::Sub(%1017, %1031) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:190:0
  %1033 : Float(22800, 1) = onnx::Flatten[axis=1](%1032) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:190:0
  %1034 : Tensor = onnx::Shape(%1029)
  %1035 : Tensor = onnx::Constant[value={0}]()
  %1036 : Long() = onnx::Gather[axis=0](%1034, %1035)
  %1037 : Tensor = onnx::Cast[to=7](%1036)
  %1038 : Tensor = onnx::Constant[value={0}]()
  %1039 : Tensor = onnx::Constant[value={1}]()
  %1040 : Tensor = onnx::Range(%1038, %1037, %1039)
  %1041 : Tensor = onnx::Shape(%1029)
  %1042 : Tensor = onnx::Constant[value={1}]()
  %1043 : Long() = onnx::Gather[axis=0](%1041, %1042)
  %1044 : Tensor = onnx::Cast[to=7](%1043)
  %1045 : Tensor = onnx::Constant[value={0}]()
  %1046 : Tensor = onnx::Constant[value={1}]()
  %1047 : Tensor = onnx::Range(%1045, %1044, %1046)
  %1048 : Tensor = onnx::Constant[value={0}]()
  %1049 : Tensor = onnx::Constant[value={0}]()
  %1050 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1051 : Tensor = onnx::Constant[value={4}]()
  %1052 : Tensor = onnx::Slice(%1047, %1049, %1050, %1048, %1051)
  %1053 : Tensor = onnx::Constant[value=-1  1 [ CPULongType{2} ]]()
  %1054 : Tensor = onnx::Reshape(%1040, %1053)
  %1055 : Tensor = onnx::Constant[value={-1}]()
  %1056 : Tensor = onnx::Reshape(%1052, %1055)
  %1057 : Tensor = onnx::Add(%1054, %1056)
  %1058 : Tensor = onnx::Shape(%1057)
  %1059 : Tensor = onnx::Shape(%1058)
  %1060 : Tensor = onnx::ConstantOfShape[value={1}](%1059)
  %1061 : Long() = onnx::Constant[value={-1}]()
  %1062 : LongTensor = onnx::Mul(%1060, %1061)
  %1063 : Tensor = onnx::Equal(%1058, %1062)
  %1064 : Tensor = onnx::Where(%1063, %1060, %1058)
  %1065 : Tensor = onnx::Expand(%1054, %1064)
  %1066 : Tensor = onnx::Unsqueeze[axes=[-1]](%1065)
  %1067 : Tensor = onnx::Shape(%1058)
  %1068 : Tensor = onnx::ConstantOfShape[value={1}](%1067)
  %1069 : Long() = onnx::Constant[value={-1}]()
  %1070 : LongTensor = onnx::Mul(%1068, %1069)
  %1071 : Tensor = onnx::Equal(%1058, %1070)
  %1072 : Tensor = onnx::Where(%1071, %1068, %1058)
  %1073 : Tensor = onnx::Expand(%1056, %1072)
  %1074 : Tensor = onnx::Unsqueeze[axes=[-1]](%1073)
  %1075 : Tensor = onnx::Concat[axis=-1](%1066, %1074)
  %1076 : Tensor = onnx::Shape(%1029)
  %1077 : Tensor = onnx::Constant[value={0}]()
  %1078 : Tensor = onnx::Constant[value={2}]()
  %1079 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1080 : Tensor = onnx::Slice(%1076, %1078, %1079, %1077)
  %1081 : Tensor = onnx::Concat[axis=0](%1058, %1080)
  %1082 : Tensor = onnx::Reshape(%1033, %1081)
  %1083 : Float(22800, 1) = onnx::ScatterND(%1029, %1075, %1082)
  %1084 : Float() = onnx::Constant[value={0.5}]()
  %1085 : Float(22800, 1) = onnx::Mul(%1027, %1084)
  %1086 : Float(22800, 1) = onnx::Sub(%1021, %1085) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:191:0
  %1087 : Float(22800, 1) = onnx::Flatten[axis=1](%1086) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:191:0
  %1088 : Tensor = onnx::Shape(%1083)
  %1089 : Tensor = onnx::Constant[value={0}]()
  %1090 : Long() = onnx::Gather[axis=0](%1088, %1089)
  %1091 : Tensor = onnx::Cast[to=7](%1090)
  %1092 : Tensor = onnx::Constant[value={0}]()
  %1093 : Tensor = onnx::Constant[value={1}]()
  %1094 : Tensor = onnx::Range(%1092, %1091, %1093)
  %1095 : Tensor = onnx::Shape(%1083)
  %1096 : Tensor = onnx::Constant[value={1}]()
  %1097 : Long() = onnx::Gather[axis=0](%1095, %1096)
  %1098 : Tensor = onnx::Cast[to=7](%1097)
  %1099 : Tensor = onnx::Constant[value={0}]()
  %1100 : Tensor = onnx::Constant[value={1}]()
  %1101 : Tensor = onnx::Range(%1099, %1098, %1100)
  %1102 : Tensor = onnx::Constant[value={0}]()
  %1103 : Tensor = onnx::Constant[value={1}]()
  %1104 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1105 : Tensor = onnx::Constant[value={4}]()
  %1106 : Tensor = onnx::Slice(%1101, %1103, %1104, %1102, %1105)
  %1107 : Tensor = onnx::Constant[value=-1  1 [ CPULongType{2} ]]()
  %1108 : Tensor = onnx::Reshape(%1094, %1107)
  %1109 : Tensor = onnx::Constant[value={-1}]()
  %1110 : Tensor = onnx::Reshape(%1106, %1109)
  %1111 : Tensor = onnx::Add(%1108, %1110)
  %1112 : Tensor = onnx::Shape(%1111)
  %1113 : Tensor = onnx::Shape(%1112)
  %1114 : Tensor = onnx::ConstantOfShape[value={1}](%1113)
  %1115 : Long() = onnx::Constant[value={-1}]()
  %1116 : LongTensor = onnx::Mul(%1114, %1115)
  %1117 : Tensor = onnx::Equal(%1112, %1116)
  %1118 : Tensor = onnx::Where(%1117, %1114, %1112)
  %1119 : Tensor = onnx::Expand(%1108, %1118)
  %1120 : Tensor = onnx::Unsqueeze[axes=[-1]](%1119)
  %1121 : Tensor = onnx::Shape(%1112)
  %1122 : Tensor = onnx::ConstantOfShape[value={1}](%1121)
  %1123 : Long() = onnx::Constant[value={-1}]()
  %1124 : LongTensor = onnx::Mul(%1122, %1123)
  %1125 : Tensor = onnx::Equal(%1112, %1124)
  %1126 : Tensor = onnx::Where(%1125, %1122, %1112)
  %1127 : Tensor = onnx::Expand(%1110, %1126)
  %1128 : Tensor = onnx::Unsqueeze[axes=[-1]](%1127)
  %1129 : Tensor = onnx::Concat[axis=-1](%1120, %1128)
  %1130 : Tensor = onnx::Shape(%1083)
  %1131 : Tensor = onnx::Constant[value={0}]()
  %1132 : Tensor = onnx::Constant[value={2}]()
  %1133 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1134 : Tensor = onnx::Slice(%1130, %1132, %1133, %1131)
  %1135 : Tensor = onnx::Concat[axis=0](%1112, %1134)
  %1136 : Tensor = onnx::Reshape(%1087, %1135)
  %1137 : Float(22800, 1) = onnx::ScatterND(%1083, %1129, %1136)
  %1138 : Float() = onnx::Constant[value={0.5}]()
  %1139 : Float(22800, 1) = onnx::Mul(%1024, %1138)
  %1140 : Float(22800, 1) = onnx::Add(%1017, %1139) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:192:0
  %1141 : Float(22800, 1) = onnx::Flatten[axis=1](%1140) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:192:0
  %1142 : Tensor = onnx::Shape(%1137)
  %1143 : Tensor = onnx::Constant[value={0}]()
  %1144 : Long() = onnx::Gather[axis=0](%1142, %1143)
  %1145 : Tensor = onnx::Cast[to=7](%1144)
  %1146 : Tensor = onnx::Constant[value={0}]()
  %1147 : Tensor = onnx::Constant[value={1}]()
  %1148 : Tensor = onnx::Range(%1146, %1145, %1147)
  %1149 : Tensor = onnx::Shape(%1137)
  %1150 : Tensor = onnx::Constant[value={1}]()
  %1151 : Long() = onnx::Gather[axis=0](%1149, %1150)
  %1152 : Tensor = onnx::Cast[to=7](%1151)
  %1153 : Tensor = onnx::Constant[value={0}]()
  %1154 : Tensor = onnx::Constant[value={1}]()
  %1155 : Tensor = onnx::Range(%1153, %1152, %1154)
  %1156 : Tensor = onnx::Constant[value={0}]()
  %1157 : Tensor = onnx::Constant[value={2}]()
  %1158 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1159 : Tensor = onnx::Constant[value={4}]()
  %1160 : Tensor = onnx::Slice(%1155, %1157, %1158, %1156, %1159)
  %1161 : Tensor = onnx::Constant[value=-1  1 [ CPULongType{2} ]]()
  %1162 : Tensor = onnx::Reshape(%1148, %1161)
  %1163 : Tensor = onnx::Constant[value={-1}]()
  %1164 : Tensor = onnx::Reshape(%1160, %1163)
  %1165 : Tensor = onnx::Add(%1162, %1164)
  %1166 : Tensor = onnx::Shape(%1165)
  %1167 : Tensor = onnx::Shape(%1166)
  %1168 : Tensor = onnx::ConstantOfShape[value={1}](%1167)
  %1169 : Long() = onnx::Constant[value={-1}]()
  %1170 : LongTensor = onnx::Mul(%1168, %1169)
  %1171 : Tensor = onnx::Equal(%1166, %1170)
  %1172 : Tensor = onnx::Where(%1171, %1168, %1166)
  %1173 : Tensor = onnx::Expand(%1162, %1172)
  %1174 : Tensor = onnx::Unsqueeze[axes=[-1]](%1173)
  %1175 : Tensor = onnx::Shape(%1166)
  %1176 : Tensor = onnx::ConstantOfShape[value={1}](%1175)
  %1177 : Long() = onnx::Constant[value={-1}]()
  %1178 : LongTensor = onnx::Mul(%1176, %1177)
  %1179 : Tensor = onnx::Equal(%1166, %1178)
  %1180 : Tensor = onnx::Where(%1179, %1176, %1166)
  %1181 : Tensor = onnx::Expand(%1164, %1180)
  %1182 : Tensor = onnx::Unsqueeze[axes=[-1]](%1181)
  %1183 : Tensor = onnx::Concat[axis=-1](%1174, %1182)
  %1184 : Tensor = onnx::Shape(%1137)
  %1185 : Tensor = onnx::Constant[value={0}]()
  %1186 : Tensor = onnx::Constant[value={2}]()
  %1187 : Tensor = onnx::Constant[value={9223372036854775807}]()
  %1188 : Tensor = onnx::Slice(%1184, %1186, %1187, %1185)
  %1189 : Tensor = onnx::Concat[axis=0](%1166, %1188)
  %1190 : Tensor = onnx::Reshape(%1141, %1189)
  %1191 : Float(22800, 1) = onnx::ScatterND(%1137, %1183, %1190)
  %1192 : Float() = onnx::Constant[value={0.5}]()
  %1193 : Float(22800, 1) = onnx::Mul(%1027, %1192)
  %1194 : Float(22800, 1) = onnx::Add(%1021, %1193) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:193:0
  %1195 : Float(22800, 1) = onnx::Flatten[axis=1](%1194) # /media/WorkSpace/Development_Repository/Working/models/boa/box_regression.py:193:0
  %1196 : Tensor = onnx::Shape(%1191)

The output model
What could be the reason for such an issue? The tract logs show that the data is not empty. But the exported model (as shown in the figure) has empty data. Could somebody give me some inputs to locate the problem? Thanks in advance.


GPU memory shoot up while using cuda11.3

Question:
We are using NER model. Our model was trained using torch 1.6. It took around 1.9 GB of RAM. We have moved from Tesla V100 to A100. As A100 can be run only on Cuda>=11.0 we have been forced to install the latest PyTorch with 11.3 support. While doing this, the memory needed for our model increased from 1.9 GB to 3.2 GB.
Could anyone shed some idea here on why such a huge difference and what can be done to reduce this?  Will retraining model using torch 1.10 reduce memory usage?
I am attaching stats on different version and their memory usage. Thanks for your help.

Screenshot 2021-12-09 at 5.40.25 PM1380×1568 34.9 KB

Context:
I don’t know how you’ve measured the GPU memory usage and also don’t know if you are facing any issues (e.g. if you had to lower the batch size or change the training in any way).
Some expected reasons for a different memory usage:

the CUDA context size is expected to increase due to the additional SMs on A100 compared to V100
besides that, also the libraries increased in size (CUDA, cuDNN etc.)
If you are checking the peak memory, note that cuDNN could pick kernels with a larger workspace on A100, if they are not yielding OOM errors and are giving a speedup (especially if cudnn.benchmark=True is used). You can limit the cuDNN workspace via CUDNN_CONV_WSCAP_DBG


“Illegal instruction” using AutoModelWithLMHead on 64bit Raspberry Pi 4B

Question:
from transformers import AutoModelWithLMHead
model = AutoModelWithLMHead.from_pretrained('fine-tuned-DialoGPT')
model.generate(...)

This my code, and I’m running this on a 64-bit Raspberry Pi 4B. It crashes when model.generate(...) is executed, and only prints out Illegal instruction. It works fine on my computer though.
What may be causing this issue?

Context:
Update: I’ve tried using https://github.com/KumaTea/pytorch-aarch64 2 and I got a response with eatures mathemat mathemat mathemat mathemat mathemat (repeat this 1000 times)........ It took around 5 minutes to generate that response.
I still don’t know how to use pytorch on a 64-bit RasPi 4B. Help is appreciated.


Question:
Hi all!
Does Pytorch supports CUDA 11.4 toolkit? Or could I build pytorch with cuda 11.4 support?

Context: Yes, you can build PyTorch from source using all released CUDA versions between 10.2 and 11.5.


Need Suggestions for upgrading older version pytorch

Question:
Hi guys, my model can only work with pytorch <=1.1.0, unfortunately pytorch 1.1.0 can’t work with CUDA 11, which is needed in new GPU card like Nvidia A100 and 30xx series.
When I tried to use pytorch 1.7 + CUDA 11, my trained model will produce strange noise in output images.  Model trained on pytorch 1.1.0 is OK, models trained on pytorch 1.1.0 can work in a forward loop in pytorch 1.7+CUDA11 environment.
My question is,

If I want to upgrade to pytorch 1.7, is there any guideline how to upgrade from pytorch 1.1.0 to pytorch 1.7, since there are many upgraded functions between versions. I don’t know which function I should pay attention to.
If I want to use pytorch 1.1.0 with CUDA 11 in new GPU card, is possible, and how to?

Thanks for the attention~

Context:
You could skim through the release notes for each released PyTorch version and check which changes might affect your training. I would probably start with backwards compatibility breaking changes and see, if fixing these might help.

You would have to search for all CUDA 11.x related PRs and try to backport them to the older PyTorch release, which might be cumbersome depending how old the PyTorch release is (and might not be trivial at all).


TorchServe: MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding

Question:
While serving the model on Docker, i get the following error when it tried to load the saved model.
Strangely it cant also not find torch although it is install.
2021-11-25 20:18:12,416 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2021-11-25 20:18:12,438 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: ponzi.mar
2021-11-25 20:18:12,455 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 0.1 for model ponzi
2021-11-25 20:18:12,456 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 0.1 for model ponzi
2021-11-25 20:18:12,456 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model ponzi loaded.
2021-11-25 20:18:12,456 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: ponzi, count: 12
2021-11-25 20:18:12,514 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2021-11-25 20:18:12,683 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,684 [INFO ] W-9005-ponzi_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-ponzi_0.1-stdout
2021-11-25 20:18:12,683 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,687 [INFO ] W-9003-ponzi_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9003-ponzi_0.1-stdout
2021-11-25 20:18:12,683 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,683 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,689 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,689 [WARN ] W-9005-ponzi_0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-ponzi_0.1-stderr
2021-11-25 20:18:12,683 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,689 [WARN ] W-9005-ponzi_0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9005-ponzi_0.1-stdout
2021-11-25 20:18:12,690 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,690 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,683 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,683 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,690 [WARN ] W-9007-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,689 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,689 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,688 [WARN ] W-9005-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,690 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,690 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,690 [WARN ] W-9003-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,684 [INFO ] W-9006-ponzi_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9006-ponzi_0.1-stdout
2021-11-25 20:18:12,683 [WARN ] W-9000-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,683 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,690 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,690 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,690 [INFO ] W-9005-ponzi_0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9005-ponzi_0.1-stderr
2021-11-25 20:18:12,690 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,691 [WARN ] W-9004-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,691 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,691 [WARN ] W-9001-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,689 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,692 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,692 [WARN ] W-9009-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,689 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,683 [WARN ] W-9008-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,684 [WARN ] W-9011-ponzi_0.1-stderr MODEL_LOG - Fatal Python error: Py_Initialize: Unable to get the locale encoding
2021-11-25 20:18:12,692 [WARN ] W-9010-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,691 [WARN ] W-9002-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'
2021-11-25 20:18:12,691 [WARN ] W-9003-ponzi_0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9003-ponzi_0.1-stderr
2021-11-25 20:18:12,691 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG - Traceback (most recent call last):
2021-11-25 20:18:12,692 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG -   File "/home/model-server/tmp/models/d272938c278d47eb9c045c2a929e667b/encodings.py", line 1, in <module>
2021-11-25 20:18:12,693 [WARN ] W-9006-ponzi_0.1-stderr MODEL_LOG - ModuleNotFoundError: No module named 'torch'

The Docker file setup is like this.
FROM pytorch/torchserve:latest

USER root
RUN printf "\nservice_envelope=json" >> /home/model-server/config.properties \
    && pip install pytorch-lightning torchmetrics google-cloud-storage scikit-learn
USER model-server

COPY src /home/model-server/src
COPY load_model.py /home/model-server/src
COPY configs /home/model-server/configs
COPY /models/state_dict.pt /home/model-server/src/
RUN mkdir "model_store"

RUN torch-model-archiver \
    --model-name ponzi \
    --version 0.1 \
    --model-file /home/model-server/src/model.py \
    --serialized-file /home/model-server/src/state_dict.pt \
    --handler /home/model-server/src/handler.py \
    --export-path /home/model-server/model_store \
    --extra-files /home/model-server/configs/features.yml,/home/model-server/configs/model_configs.yml,/home/model-server/src/utils.py,/home/model-server/src/encodings.py,/home/model-server/src/dataset.py,/home/model-server/src/__init__.py

CMD ["torchserve", \
     "--start", \
     "--ncs", \
     "--model-store model_store", \
     "--ts-config=/home/model-server/config.properties", \
     "--models ponzi=ponzi.mar"]

Can someone help here? Thanks.

Context:
Based on these error messages:
Fatal Python error: Py_Initialize: Unable to get the locale encoding
[...]
ModuleNotFoundError: No module named 'torch'

I would guess that your current setup might be mixing different Python versions (or environments) and thus breaks in Py_Initialize and cannot find torch.
Look through the build logs of your container and check if some pre-installed packages are uninstalled etc.


Question:
I have a weird problem. I can load the state_dict in my machine which was trained with Pytorch lightning on the GPU. However if i push it to docker it does not load. The model is a Pytorch Lightning module.

On Machine
loaded_model = LitModel()
loaded_model.load_state_dict(torch.load(‘models/trained-model-2021.pt’), strict=False)
Parameter containing:
tensor([[ 0.0817, -0.2579, -0.0080,  …,  0.1619, -0.0783, -0.0087],
[ 0.0866, -0.2708,  0.2528,  …,  0.0203, -0.2319,  0.1011],
[ 0.0564,  0.0195, -0.2858,  …, -0.1899,  0.1004, -0.1013],
…,
[ 0.2584, -0.2400, -0.0358,  …, -0.1391,  0.2278,  0.0264],
[ 0.1901, -0.3056, -0.2537,  …, -0.0747,  0.2703,  0.0803],
[-0.2639, -0.0381,  0.0538,  …,  0.2504,  0.2152,  0.1084]],
requires_grad=True)

On Docker
device = torch.device(‘cpu’)
loaded_model = LitModel()
loaded_model.load_state_dict(torch.load(‘models/trained-model-2021.pt’, map_location=device), strict=False)
loaded_model.load_state_dict(torch.load(‘src/trained-model-2021-11-23–23:08-hs.pt’, map_location=device), strict=True)
Traceback (most recent call last):
File “”, line 1, in
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 1407, in load_state_dict
self.class.name, “\n\t”.join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for LitModel:
Missing key(s) in state_dict: “layer_1.weight”, “layer_1.bias”, “layer_3.weight”, “layer_3.bias”.
Unexpected key(s) in state_dict: “model”.
Any ideas what could be wrong here.

Context:
Based on the error message it seems you are not loading a state_dict, but a custom dict class containing the 'model' key, so you might want to access it first.
Something like this could work:
loaded_model.load_state_dict(torch.load(‘src/trained-model-2021-11-23–23:08-hs.pt’, map_location=device)['model'])


Why is torch-1.10.0+cu113 much larger than torch-1.10.0

Question: I am deploying our ML models in a docker container, and we try to reduce the size of this docker image.  I notice that torch-1.10.0+cu113 is more than 1GB larger than torch-1.10.0. The main difference is caused by torch/lib/libtorch_cuda_cpp.so, which only exists in torch-1.10.0+cu113. What’s the use of this file and can we not have it but still make it works with CUDA 11.3?

Context: The GPU kernels are separate in PyTorch precisely for the reason you’re describing: they add a substantial amount of footprint to the library size. This file contains all the CUDA kernels for GPU execution; without them there wouldn’t be any native CUDA support.


Compile PyTorch with MPI as conda package

Question:
What I am trying
I am trying to compile PyTorch with MPI, CUDA and CUDNN support and package it as a conda package.
As a first step I just want to compile against an arbitrary IntelMPI and CUDA/CUDNN version and get a working package.
Unfortunately there are not many good guides out there that explain the process.
My questions

Do you know any good guide which explains how to compile pytorch with mpi, cuda and cudnn support and package it with conda?
Where are the recipes used by the pytorchbot, which uploads the pytorch packages to anaconda?
How do I fix these WARNING (pytorch,lib/python3.8/site-packages/torch/test/cpuid_test): $RPATH/libtorch_cuda.so not found in packages, sysroot(s) nor the missing_dso_whitelist.  errors?
Why does the package process abort? I have no clue why it does not succeed.
Is it normal that the compilation of CUDA NVCC uses just one core?

Status quo
The compilation ends successfully, the packaging not.
IntelMPI, CUDNN and CUDA are all found correctly in the configure process (see install.log)
The meta.yaml looks like this (I know that I dont need all of these packages at runtime, but I just want to make it work as a first step):
package:
  name: pytorch
  version: 1.10.0

source:
  git_rev: v1.10.0
  git_url: https://github.com/pytorch/pytorch.git

requirements:
  build:
    - python
    - setuptools
    - astunparse
    - numpy
    - ninja
    - pyyaml
    - mkl
    - mkl-include
    - setuptools
    - cmake
    - cffi
    - typing_extensions
    - future
    - six
    - requests
    - dataclasses

  run:
    - python
    - astunparse
    - numpy
    - ninja
    - pyyaml
    - mkl
    - mkl-include
    - setuptools
    - cmake
    - cffi
    - typing_extensions
    - future
    - six
    - requests
    - dataclasses

  host:
    - python
    - astunparse
    - numpy
    - ninja
    - pyyaml
    - mkl
    - mkl-include
    - setuptools
    - cmake
    - cffi
    - typing_extensions
    - future
    - six
    - requests
    - dataclasses

The build.sh looks like this:
#!/usr/bin/env bash

set -x
set -e
set -u

$PYTHON setup.py clean 2>&1
USE_NINJA=0 USE_CUDA=1 USE_CUDNN=1 USE_NUMPY=1 USE_DISTRIBUTED=1 USE_MPI=1 USE_GLOO=1 USE_SYSTEM_NCCL=0 MAX_JOBS=$(nproc) MAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} $PYTHON -u setup.py install --cmake 2>&1
echo "Successfully biuld pytorch"

and I run conda-build like this
conda-build . --keep-old-work --dirty |& tee install.log

The important parts of the  install.log are:
## Package Plan ##

  environment location: /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_build_env

The following NEW packages will be INSTALLED:

    _libgcc_mutex:      0.1-main
    _openmp_mutex:      4.5-1_gnu
    astunparse:         1.6.3-py_0
    blas:               1.0-mkl
    brotlipy:           0.7.0-py38h27cfd23_1003
    bzip2:              1.0.8-h7b6447c_0
    c-ares:             1.17.1-h27cfd23_0
    ca-certificates:    2021.10.26-h06a4308_2
    certifi:            2021.10.8-py38h06a4308_0
    cffi:               1.14.6-py38h400218f_0
    charset-normalizer: 2.0.4-pyhd3eb1b0_0
    cmake:              3.19.6-h973ab73_0
    cryptography:       35.0.0-py38hd23ed53_0
    dataclasses:        0.8-pyh6d0b6a4_7
    expat:              2.4.1-h2531618_2
    future:             0.18.2-py38_1
    idna:               3.2-pyhd3eb1b0_0
    intel-openmp:       2021.4.0-h06a4308_3561
    krb5:               1.19.2-hac12032_0
    ld_impl_linux-64:   2.35.1-h7274673_9
    libcurl:            7.78.0-h0b77cf5_0
    libedit:            3.1.20210910-h7f8727e_0
    libev:              4.33-h7f8727e_1
    libffi:             3.3-he6710b0_2
    libgcc-ng:          9.3.0-h5101ec6_17
    libgfortran-ng:     7.5.0-ha8ba4b0_17
    libgfortran4:       7.5.0-ha8ba4b0_17
    libgomp:            9.3.0-h5101ec6_17
    libnghttp2:         1.41.0-hf8bcb03_2
    libssh2:            1.9.0-h1ba5d50_1
    libstdcxx-ng:       9.3.0-hd4cf53a_17
    libuv:              1.40.0-h7b6447c_0
    lz4-c:              1.9.3-h295c915_1
    mkl:                2021.4.0-h06a4308_640
    mkl-include:        2021.4.0-h06a4308_640
    mkl-service:        2.4.0-py38h7f8727e_0
    mkl_fft:            1.3.1-py38hd3c417c_0
    mkl_random:         1.2.2-py38h51133e4_0
    ncurses:            6.3-h7f8727e_2
    ninja:              1.10.2-py38hd09550d_3
    numpy:              1.16.6-py38h2d18471_3
    numpy-base:         1.16.6-py38hdc34a94_3
    openssl:            1.1.1l-h7f8727e_0
    pip:                21.2.4-py38h06a4308_0
    pycparser:          2.21-pyhd3eb1b0_0
    pyopenssl:          21.0.0-pyhd3eb1b0_1
    pysocks:            1.7.1-py38h06a4308_0
    python:             3.8.12-h12debd9_0
    pyyaml:             6.0-py38h7f8727e_1
    readline:           8.1-h27cfd23_0
    requests:           2.26.0-pyhd3eb1b0_0
    rhash:              1.4.1-h3c74f83_1
    setuptools:         58.0.4-py38h06a4308_0
    six:                1.16.0-pyhd3eb1b0_0
    sqlite:             3.36.0-hc218d9a_0
    tk:                 8.6.11-h1ccaba5_0
    typing_extensions:  3.10.0.2-pyh06a4308_0
    urllib3:            1.26.7-pyhd3eb1b0_0
    wheel:              0.37.0-pyhd3eb1b0_1
    xz:                 5.2.5-h7b6447c_0
    yaml:               0.2.5-h7b6447c_0
    zlib:               1.2.11-h7b6447c_3
    zstd:               1.4.9-haebb681_0

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
+ set -e
+ set -u
+ /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/python setup.py clean
+ USE_NINJA=0
+ USE_CUDA=1
+ USE_CUDNN=1
+ USE_NUMPY=1
+ USE_DISTRIBUTED=1
+ USE_MPI=1
+ USE_GLOO=1
+ USE_SYSTEM_NCCL=0
++ nproc
+ MAX_JOBS=64
+ MAKE_PREFIX_PATH=/anaconda/envs/compile/conda-bld/pytorch_1636708344734/_build_env
+ /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/python -u setup.py install --cmake
source tree in: /anaconda/envs/compile/conda-bld/pytorch_1636708344734/work
export PREFIX=/anaconda/envs/compile/conda-bld/pytorch_1636708344734

--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;IDEEP_USE_MKL;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS
--   CMAKE_PREFIX_PATH     : $PREFIX/lib/python3.8/site-packages;/usr/local/cuda
--   CMAKE_INSTALL_PREFIX  : $SRC_DIR/torch
--   USE_GOLD_LINKER       : OFF
--
--   TORCH_VERSION         : 1.10.0
--   CAFFE2_VERSION        : 1.10.0
--   BUILD_CAFFE2          : ON
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_CAFFE2_MOBILE   : OFF
--   BUILD_STATIC_RUNTIME_BENCHMARK: OFF
--   BUILD_TENSOREXPR_BENCHMARK: OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.8.12
--     Python executable   : $PREFIX/bin/python
--     Pythonlibs version  : 3.8.12
--     Python library      : $PREFIX/lib/libpython3.8.so.1.0
--     Python includes     : $PREFIX/include/python3.8
--     Python site-packages: lib/python3.8/site-packages
--   BUILD_SHARED_LIBS     : ON
--   CAFFE2_USE_MSVC_STATIC_RUNTIME     : OFF
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   BUILD_MOBILE_AUTOGRAD : OFF
--   BUILD_LITE_INTERPRETER: OFF
--   INTERN_BUILD_MOBILE   :
--   USE_BLAS              : 1
--     BLAS                : mkl
--   USE_LAPACK            : 1
--     LAPACK              : mkl
--   USE_ASAN              : OFF
--   USE_CPP_CODE_COVERAGE : OFF
--   USE_CUDA              : 1
--     Split CUDA          : OFF
--     CUDA static link    : OFF
--     USE_CUDNN           : 1
--     USE_EXPERIMENTAL_CUDNN_V8_API: OFF
--     CUDA version        : 11.1
--     cuDNN version       : 8.1.1
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
--     cudart library      : /usr/local/cuda/lib64/libcudart.so
--     cublas library      : /usr/local/cuda/lib64/libcublas.so
--     cufft library       : /usr/local/cuda/lib64/libcufft.so
--     curand library      : /usr/local/cuda/lib64/libcurand.so
--     cuDNN library       : /usr/lib/x86_64-linux-gnu/libcudnn.so
--     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     NVCC flags          : -Xfatbin;-compress-all;-DONNX_NAMESPACE=onnx_torch;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86;-Xcudafe;--diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl;-std=c++14;-Xcompiler;-fPIC;--expt-relaxed-constexpr;--expt-extended-lambda;-Wno-deprecated-gpu-targets;--expt-extended-lambda;-Xcompiler;-fPIC;-DCUDA_HAS_FP16=1;-D__CUDA_NO_HALF_OPERATORS__;-D__CUDA_NO_HALF_CONVERSIONS__;-D__CUDA_NO_BFLOAT16_CONVERSIONS__;-D__CUDA_NO_HALF2_OPERATORS__
--     CUDA host compiler  : /usr/bin/cc
--     NVCC --device-c     : OFF
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    :
--   USE_FBGEMM            : ON
--     USE_FAKELOWP          : OFF
--   USE_KINETO            : ON
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_PYTORCH_METAL     : OFF
--   USE_PYTORCH_METAL_EXPORT     : OFF
--   USE_FFTW              : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : ON
--   USE_MKLDNN_ACL        : OFF
--   USE_MKLDNN_CBLAS      : OFF
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : 0
--   USE_NNPACK            : ON
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_VULKAN            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_PYTORCH_QNNPACK   : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : 1
--     USE_MPI               : 1
--     USE_GLOO              : 1
--     USE_GLOO_WITH_OPENSSL : OFF
--     USE_TENSORPIPE        : ON
--   USE_DEPLOY           : OFF
--   USE_BREAKPAD         : ON
--   Public Dependencies  : Threads::Threads;caffe2::mkl;caffe2::mkldnn
--   Private Dependencies : pthreadpool;cpuinfo;qnnpack;pytorch_qnnpack;nnpack;XNNPACK;fbgemm;/usr/lib/x86_64-linux-gnu/libnuma.so;fp16;/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/libmpicxx.so;/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/release_mt/libmpi.so;/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/libmpigi.a;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/x86_64-linux-gnu/libpthread.so;gloo;tensorpipe;aten_op_header_gen;foxi_loader;rt;fmt::fmt-header-only;kineto;gcc_s;gcc;dl
--   USE_COREML_DELEGATE     : OFF
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_PREFIX
    USE_NINJA

-- Build files have been written to: $SRC_DIR/build
cmake --build . --target install --config Release -- -j 64
Scanning dependencies of target nccl_external
Scanning dependencies of target defs.bzl
Scanning dependencies of target clog
Scanning dependencies of target libkineto_defs.bzl
Scanning dependencies of target ATEN_CPU_FILES_GEN_TARGET
Scanning dependencies of target nvfuser_rt_fp16_support
Scanning dependencies of target ATEN_CUDA_FILES_GEN_TARGET
Scanning dependencies of target mkdisp
Scanning dependencies of target nvfuser_rt_tensor
Scanning dependencies of target torch_global_deps
Scanning dependencies of target nvfuser_rt_block_reduction
Scanning dependencies of target nvfuser_rt_random_numbers
Scanning dependencies of target foxi_loader
Scanning dependencies of target mkrename
Scanning dependencies of target pthreadpool
Scanning dependencies of target nvfuser_rt_helpers
Scanning dependencies of target nvfuser_rt_grid_reduction
Scanning dependencies of target mkalias
Scanning dependencies of target nvfuser_rt_UnpackRaw
Scanning dependencies of target nvfuser_rt_PhiloxCudaStateRaw
Scanning dependencies of target gen_torch_version
Scanning dependencies of target nvfuser_rt_broadcast
[  0%] Creating directories for 'nccl_external'
Scanning dependencies of target gtest
Scanning dependencies of target common
Scanning dependencies of target mkrename_gnuabi
Scanning dependencies of target mkmasked_gnuabi
Scanning dependencies of target arraymap
[  0%] Built target defs.bzl

[  0%] Generating ../aten/src/ATen/CPUFunctions.h, ../aten/src/ATen/CPUFunctions_inl.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions_inl.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions_inl.h, ../aten/src/ATen/Declarations.yaml, ../aten/src/ATen/Functions.cpp, ../aten/src/ATen/Functions.h, ../aten/src/ATen/MetaFunctions.h, ../aten/src/ATen/MetaFunctions_inl.h, ../aten/src/ATen/NativeFunctions.h, ../aten/src/ATen/NativeMetaFunctions.h, ../aten/src/ATen/Operators.h, ../aten/src/ATen/Operators_0.cpp, ../aten/src/ATen/Operators_1.cpp, ../aten/src/ATen/Operators_2.cpp, ../aten/src/ATen/Operators_3.cpp, ../aten/src/ATen/Operators_4.cpp, ../aten/src/ATen/RedispatchFunctions.h, ../aten/src/ATen/RegisterBackendSelect.cpp, ../aten/src/ATen/RegisterCPU.cpp, ../aten/src/ATen/RegisterCompositeExplicitAutograd.cpp, ../aten/src/ATen/RegisterCompositeImplicitAutograd.cpp, ../aten/src/ATen/RegisterMeta.cpp, ../aten/src/ATen/RegisterMkldnnCPU.cpp, ../aten/src/ATen/RegisterQuantizedCPU.cpp, ../aten/src/ATen/RegisterSchema.cpp, ../aten/src/ATen/RegisterSparseCPU.cpp, ../aten/src/ATen/RegisterSparseCsrCPU.cpp, ../aten/src/ATen/RegistrationDeclarations.h, ../aten/src/ATen/CUDAFunctions.h, ../aten/src/ATen/CUDAFunctions_inl.h, ../aten/src/ATen/RegisterCUDA.cpp, ../aten/src/ATen/RegisterQuantizedCUDA.cpp, ../aten/src/ATen/RegisterSparseCUDA.cpp, ../aten/src/ATen/RegisterSparseCsrCUDA.cpp, ../aten/src/ATen/core/ATenOpList.cpp, ../aten/src/ATen/core/TensorBody.h, ../aten/src/ATen/core/TensorMethods.cpp
Scanning dependencies of target fmt
[  0%] Building C object confu-deps/cpuinfo/deps/clog/CMakeFiles/clog.dir/src/clog.c.o
Scanning dependencies of target generate-torch-sources
Scanning dependencies of target kineto_api
[  0%] Built target libkineto_defs.bzl
Scanning dependencies of target python_copy_files
Scanning dependencies of target addSuffix
[  0%] Building C object sleef/src/libm/CMakeFiles/mkrename.dir/mkrename.c.o
[  0%] Building C object sleef/src/libm/CMakeFiles/mkdisp.dir/mkdisp.c.o
[  0%] Building C object third_party/foxi/CMakeFiles/foxi_loader.dir/foxi/onnxifi_loader.c.o
[  0%] Building C object caffe2/CMakeFiles/torch_global_deps.dir/__/torch/csrc/empty.c.o
[  0%] Generating ../aten/src/ATen/CPUFunctions.h, ../aten/src/ATen/CPUFunctions_inl.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions_inl.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions_inl.h, ../aten/src/ATen/Declarations.yaml, ../aten/src/ATen/Functions.cpp, ../aten/src/ATen/Functions.h, ../aten/src/ATen/MetaFunctions.h, ../aten/src/ATen/MetaFunctions_inl.h, ../aten/src/ATen/NativeFunctions.h, ../aten/src/ATen/NativeMetaFunctions.h, ../aten/src/ATen/Operators.h, ../aten/src/ATen/Operators_0.cpp, ../aten/src/ATen/Operators_1.cpp, ../aten/src/ATen/Operators_2.cpp, ../aten/src/ATen/Operators_3.cpp, ../aten/src/ATen/Operators_4.cpp, ../aten/src/ATen/RedispatchFunctions.h, ../aten/src/ATen/RegisterBackendSelect.cpp, ../aten/src/ATen/RegisterCPU.cpp, ../aten/src/ATen/RegisterCompositeExplicitAutograd.cpp, ../aten/src/ATen/RegisterCompositeImplicitAutograd.cpp, ../aten/src/ATen/RegisterMeta.cpp, ../aten/src/ATen/RegisterMkldnnCPU.cpp, ../aten/src/ATen/RegisterQuantizedCPU.cpp, ../aten/src/ATen/RegisterSchema.cpp, ../aten/src/ATen/RegisterSparseCPU.cpp, ../aten/src/ATen/RegisterSparseCsrCPU.cpp, ../aten/src/ATen/RegistrationDeclarations.h, ../aten/src/ATen/CUDAFunctions.h, ../aten/src/ATen/CUDAFunctions_inl.h, ../aten/src/ATen/RegisterCUDA.cpp, ../aten/src/ATen/RegisterQuantizedCUDA.cpp, ../aten/src/ATen/RegisterSparseCUDA.cpp, ../aten/src/ATen/RegisterSparseCsrCUDA.cpp, ../aten/src/ATen/core/ATenOpList.cpp, ../aten/src/ATen/core/TensorBody.h, ../aten/src/ATen/core/TensorMethods.cpp
[  0%] Building CXX object third_party/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[100%] Built target caffe2_detectron_ops_gpu
Install the project...
-- Install configuration: "Release"
-- Set runtime path of "$SRC_DIR/torch/bin/protoc-3.13.0.0" to "$ORIGIN/../lib"
-- Set runtime path of "$SRC_DIR/torch/lib/libc10.so" to "$ORIGIN"
-- Set runtime path of "$SRC_DIR/torch/test/c10_CompileTimeFunctionPointer_test" to "$ORIGIN"
creating build/lib.linux-x86_64-3.8/torch/utils
copying torch/utils/throughput_benchmark.py -> build/lib.linux-x86_64-3.8/torch/utils
copying torch/utils/mkldnn.py -> build/lib.linux-x86_64-3.8/torch/utils
copying torch/utils/mobile_optimizer.py -> build/lib.linux-x86_64-3.8/torch/utils

/intel64', '/usr/local/cuda/lib64'] disagree for /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.8/site-packages/torch/test/pow_test ::
Warning: rpath /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/release_mt is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/mpi-rt/2017.0.0/intel64/lib is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)

WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): Needed DSO lib/libstdc++.so.6 found in ['libstdcxx-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): .. but ['libstdcxx-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): $RPATH/libm.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): Needed DSO lib/libgcc_s.so.1 found in ['libgcc-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): .. but ['libgcc-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): $RPATH/libc.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libcudart.so.11.0 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libtorch.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libtorch_cpu.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libtorch_cuda.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libc10_cuda.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libc10.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): Needed DSO lib/libstdc++.so.6 found in ['libstdcxx-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): .. but ['libstdcxx-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libm.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): Needed DSO lib/libgcc_s.so.1 found in ['libgcc-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): .. but ['libgcc-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libpthread.so.0 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libc.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/graph_test): $RPATH/libtorch.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/graph_test): $RPATH/libtorch_cpu.so not

[  1%] Running gen_proto.py on onnx/onnx.in.proto
[  3%] Building C object CMakeFiles/onnxifi_dummy.dir/onnx/onnxifi_dummy.c.o
[  5%] Building C object CMakeFiles/onnxifi_loader.dir/onnx/onnxifi_loader.c.o
Processing $SRC_DIR/onnx/onnx.in.proto
Writing $SRC_DIR/.setuptools-cmake-build/onnx/onnx.proto
Writing $SRC_DIR/.setuptools-cmake-build/onnx/onnx.proto3
generating $SRC_DIR/.setuptools-cmake-build/onnx/onnx_pb.py
[  7%] Running C++ protocol buffer compiler on $SRC_DIR/.setuptools-cmake-build/onnx/onnx.proto
[  9%] Linking C static library libonnxifi_loader.a
[ 11%] Linking C shared library libonnxifi_dummy.so
[ 11%] Built target onnxifi_loader
Scanning dependencies of target onnxifi_wrapper
[ 12%] Building C object CMakeFiles/onnxifi_wrapper.dir/onnx/onnxifi_wrapper.c.o
[ 12%] Built target onnxifi_dummy
CMakeFiles/gen_onnx_proto.dir/build.make:61: recipe for target 'onnx/onnx.pb.cc' failed
CMakeFiles/Makefile2:205: recipe for target 'CMakeFiles/gen_onnx_proto.dir/all' failed
[ 14%] Linking C shared module libonnxifi.so
[ 14%] Built target onnxifi_wrapper
Makefile:129: recipe for target 'all' failed
Traceback (most recent call last):
  File "/anaconda/envs/compile/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 480, in main
    execute(sys.argv[1:])
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 469, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/build.py", line 3083, in build_tree
    packages_from_this = build(metadata, stats,
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/build.py", line 2206, in build
    utils.check_call_env(cmd, env=env, rewrite_stdout_env=rewrite_env,
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/utils.py", line 410, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/utils.py", line 390, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/anaconda/envs/compile/conda-bld/onnx_1636708558027/work/conda_build.sh']' returned non-zero exit status 1.

Any help is very appreciated,
Greets from germany

Context: The pytorch/builder 1 repository contains the build scripts for the pip wheels and conda binaries, so you could compare your setup to the official one and also use the docker container workflow for the build.


Question:
Hi, so i go to Start Locally | PyTorch
and choose my system requirements i.e the following:
Pytorch Build: Stable (1.10)
Linux
Pip
Python
CUDA Version: 11.3
The CUDA part can cause problems, as i run the following cmd on both kaggle and colab and get different CUDA version.
Kaggle
!nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Wed_Jul_22_19:09:09_PDT_2020
Cuda compilation tools, release 11.0, V11.0.221
Build cuda_11.0_bu.TC445_37.28845127_0
Google Colab :
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Mon_Oct_12_20:09:46_PDT_2020
Cuda compilation tools, release 11.1, V11.1.105
Build cuda_11.1.TC455_06.29190527_0
So how can i change the CUDA Version to match 11.3 ? Or will it mess up the Notebooks environment. Thanks

Context: You won’t be able to change the local CUDA toolkit easily. The PyTorch binaries ship with their own CUDA runtime so unless you are building PyTorch from source or a custom CUDA extension the local CUDA toolkit won’t be used.


Question:
Hello,
I am looking for a way to bin tensor values.
Say my tensor has values between -0.255 and +0.256, I am looking for a way to represent this will “less numbers”.
For example I would like to bin this range into 32 values so I will only have the values -0.255, -0.239, -0.223 and so on… (the bins are in jumps of 16 because 512/32=16)
I found torch.histc but it gives me only a histogram, this way I lose “which index was mapped to each bin”.
Thanks!

Context: I’m having the same issue. Have you found a clean solution for this?


Does a docker container need to use the exact same libcuda version of the host server?

Question:
I’m trying to run an application that uses torch and tensorflow inside a docker container but I’m getting this error message:
“kernel version 450.66.0 does not match DSO version 450.142.0 – cannot find working devices in this configuration”
The cuda-compat-11-0 version inside the container is 450.142.0 but the host driver version is 450.66.0.
As I can’t find cuda-compat-11-0 version 450.66.0, I tried other versions around it but I got the same error.
I’m not able to change the Host driver, so my question is:
Do I really have to install the exact same version on the docker container in order to make it work, or am I doing something wrong?
Please help me understand this issue. Thanks in advance!

Context:
Are you using the Nvidia Container Toolkit? If not, I think this may resolve the issue for you. Please let me know how it works out.
How to Properly Use the GPU within a Docker Container | by Jacob Solawetz | Towards Data Science 31