Question:
Hi,
Im trying to make my first CNN using pyTorch and am following online help and code already people wrote. i am trying to reproduce their results. I’m using the Kaggle Dogs Breed Dataset for this and below is the error I get. The trainloader does not return my images and labels and any attempt to get them leads in an error:
Traceback (most recent call last):
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\pydevd.py", line 1664, in <module>
    main()
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "E:\Program Files\JetBrains\PyCharm Community Edition 2018.2.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/sbzfk/PycharmProjects/my_FCN_attempt/Kaggle_Dogs_Competition.py", line 85, in <module>
    img, label = next(iter(train_loader))
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 314, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 314, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "C:/Users/sbzfk/PycharmProjects/my_FCN_attempt/Kaggle_Dogs_Competition.py", line 42, in __getitem__
    img = self.transform(img)
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\transforms.py", line 34, in __call__
    img = t(img)
  File "C:\Users\sbzfk\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\transforms.py", line 187, in __call__
    w, h = img.size
TypeError: cannot unpack non-iterable int object

Below is my code:
class DogsDataset(Dataset):

    def __init__(self, filenames, labels, root_dir, transform=None):
        assert len(filenames) == len(labels)        # if the two are not of equal length throw an error
        self.filenames = filenames
        self.labels = labels
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.filenames)

    def __getitem__(self, idx):
        this_img = join(self.root_dir, 'train', self.filenames[idx]+'.jpg')
        print(this_img)
        img = io.imread(this_img)
        label = self.labels[idx]
        print(label)

        if self.transform:
            img = self.transform(img)

        return [img, label]

batch_size = 64
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

dataset_root = expanduser(join('~', 'Documents', 'kaggle_dogs_dataset'))
# join will intelligently join directories irrespective of OS, and expanduser will
# replace with /home/ in linux or the username in Windows

csv_file = pd.read_csv(join(dataset_root, 'labels.csv'))        # csv file has two columns, id which are filenames and breed which are labels
filenames = csv_file.id.values              # convert that column to an array, id is the column name and values converty to numpy array
# le = LabelEncoder()
# labels = le.fit_transform(csv_file.breed)           # this will just encode the names between 0 to models-1 , basically changing strings to integers
labels = csv_file.breed.values

filenames_train, filenames_eval, labels_train, labels_eval = train_test_split(filenames, labels,
                                                                              test_size=0.1, stratify=labels)       # this is an import from sklearn as the name implies, it randomly splits data into train and eval, 10% of it to test and rest train

data_transform = transforms.Compose([transforms.Scale(224),
                                     transforms.CenterCrop(224),
                                     transforms.ToTensor(),
                                     transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225))])

dog_train = DogsDataset(filenames_train, labels_train, dataset_root, transform=data_transform)
train_loader = DataLoader(dog_train, batch_size, shuffle=True)

dog_eval = DogsDataset(filenames_eval, labels_eval, dataset_root, transform=data_transform)
eval_loader = DataLoader(dog_eval, batch_size, shuffle=True)

def im_show(axis, inp):
    """Denormalize and show"""
    inp = inp.numpy().transpose((1, 2, 0))
    mean = np.array([0.485, 0.456, 0.406])
    std = np.array([0.229, 0.224, 0.225])
    inp = std * inp + mean
    axis.imshow(inp)

img, label = next(iter(train_loader))
print(img.size(), label.size())
fig = plt.figure(1, figsize=(16, 4))
grid = ImageGrid(fig, 111, nrows_ncols=(1, 4), axes_pad=0.05)
for i in range(img.size()[0]):
    ax = grid[i]
    im_show(ax, img[i])

Context:
I guess your self.transform contains some image transformations other than ToTensor(), e.g. RandomCrop().
These transformations currently work on PIL.Images, so if you load your image with img = Image.open(this_img) it should work.


‘Tensor’ object is not callable.how can i handle this,floks

Question:
TypeError: ‘Tensor’ object is not callable
how can i handle this,floks

Context:
Could you post a code snippet throwing this error?
I would generally recommend to use the factory method torch.tensor instead of torch.Tensor, since the latter will return uninitialized values if you provide a tensor shape.


Question:
Hello,
I´m running my model (using pytorch lightning) in a cluster with multiples GPUs (2). My problem is that I would like to access all the datapoints in the batch. Because I´m using more than 2 GPUs, my batch in divided between those two devices for parallelisation purposes, which means than when I access the data in the batch in eval/training, I´m getting just half the batch.
How could I obtain the complete batch and the predictions of the model that are divided among different devices/GPUs?
Thanks!

Context:
fermoren:

Because I´m using more than 2 GPUs, my batch in divided between those two devices for parallelisation purposes, which means than when I access the data in the batch in eval/training, I´m getting just half the batch.

Are you using nn.DataParallel or DistributedDataParallel?
In the former (not recommended) case the batch would be split while passing it to the DataParallel model, so inside the forward of the models you would only see a chunk of the data. The output however is gathered again on the default device and you will see the output for the entire batch.
In the latter case, each process would use a single GPU so would be responsible to load the entire batch.


PyTorch distributed with MPI on Multi-node Multi-GPUs

Question:
Hi,
I’m trying to run a PyTorch DDP code on 2 nodes with 8 GPUs each with mpirun. I want to use 1 mpi
rank per node to launch the DDP job per node and let DDP launch 8 worker threads in each node. The command I’m using is
let N_NODES=2
let N_RANKS=2
for (( node_rank=0; node_rank<$N_NODES; node_rank++ ))
do
        mpirun -np $N_RANKS -npernode 1 -hostfile $HOSTFILE --map-by node \
        python -u -m torch.distributed.launch  --master_addr=$master_addr --nproc_per_node=8  --nnodes=$N_NODES \
        --node_rank=$node_rank train.py &

        pids[${node_rank}]=$!
done

When I check the activity on the GPUs with nvidia-smi, I see the code running on few GPUs on both nodes (6 on first node and 4 on second node). What is missing here to run on all 8 GPUs in each node, 16 in total? Any help is kindly appreciated.
Also, if I just launch 2 ranks with 1 rank per node with the below command, I see the expected execution, one instance running on a single gpu in each node.

mpirun -np 2 -npernode 1 -hostfile $HOSTFILE --map-by node \
python -u -m torch.distributed.launch  --master_addr=$master_addr  train.py

With the following way, I see 8 process on 1st node and 16 processes on 2nd node

mpirun -np 2 -npernode 1 -hostfile $HOSTFILE --map-by node \
python -u -m torch.distributed.launch --nproc_per_node=8  train.py

Or is there another way to launch the distributed implementation across multi-node multi-gpu system with mpirun.
Thanks,
K

Context: Did you find any drop in data loader performance while running through mpirun ??


Legacy autograd function with non-static forward method - old pytorch code

Question:
Trying to run some code from here. If I run the test.py after removing the require_gradients() on line 39 (which I’m not sure how to avoid the leaf variable error otherwise for) I still receive the following error message:

RuntimeError                              Traceback (most recent call last)
Input In [23], in
61 print(f"Solving K={K} linear systems that are {n} x {n} with {As[0].nnz} nonzeros and {m} right hand sides.")
63 cg = CG(A_bmm, M_bmm=M_bmm, rtol=1e-5, atol=1e-5, verbose=True)
—> 64 X = cg(B_torch)
66 start = time.perf_counter()
67 X_np = np.concatenate([np.hstack([splinalg.cg(A, B[:, i], M=M)[0][:, np.newaxis] for i in range(m)])[np.newaxis, :, :]
68                        for A, B, M in zip(As, Bs, Ms)], 0)
File ~/.local/lib/python3.9/site-packages/torch/autograd/function.py:261, in Function.call(self, *args, **kwargs)
260 def call(self, *args, **kwargs):
 → 261     raise RuntimeError(
262         "Legacy autograd function with non-static forward method is deprecated. "
263         "Please use new-style autograd function with static forward method. "
264         “(Example: Automatic differentiation package - torch.autograd — PyTorch 1.10.1 documentation 1)”)
RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method.

I’ve read some on this forum about this error, but cannot single out the problem in the definition of the forward, backward functions:
class CG(torch.autograd.Function):
def __init__(self, A_bmm, M_bmm=None, rtol=1e-3, atol=0., maxiter=None, verbose=False):
    self.A_bmm = A_bmm
    self.M_bmm = M_bmm
    self.rtol = rtol
    self.atol = atol
    self.maxiter = maxiter
    self.verbose = verbose

def forward(self, B, X0=None):
    X, _ = cg_batch(self.A_bmm, B, M_bmm=self.M_bmm, X0=X0, rtol=self.rtol,
                 atol=self.atol, maxiter=self.maxiter, verbose=self.verbose)
    return X

def backward(self, dX):
    dB, _ = cg_batch(self.A_bmm, dX, M_bmm=self.M_bmm, rtol=self.rtol,
                  atol=self.atol, maxiter=self.maxiter, verbose=self.verbose)
    return dB

Context: The error message links to an example of a custom autograd.Function, which e.g. includes the @staticmethod decorators which are missing in your code.


Question:
I have to implement a Convolutional Neural Network, that takes a kinect image (1640480) and return a 1 x8 tensor predicting the class to which the object belongs and a 1 x 4 tensor, predicting the bounding box around the image, if its present.
Please help me how can I implement a suitable model to give two outputs and how to calculate loss and backpropagate in that case?
Also, i have just around 6000 training images, how can I achieve the best possible results with limited number of training images??
Some examples of such projects would be highly helpful.

Context:
Multiple outputs is pretty straightforward. Just return mutiple values in the forward() method of your net.
def forward(self, x):
    # Do your stuff here
    ...
    x1 = F.log_softmax(x) # class probabilities
    x2 = ... # bounding box calculation
    return x1, x2

Using these two outputs, you can define two different loss functions and just add them.
out1, out2 = model(data)
loss1 = criterion1(out1, target1)
loss2 = criterion2(out2, target2)
loss = loss1 + loss2
loss.backward()


RuntimeError: mat1 and mat2 shapes cannot be multiplied (64x13056 and 153600x2048)

Question:
who can help me?
import numpy as np
from PIL import Image
import glob
import torch
import torch.nn as nn
import torch.optim
from torch.autograd import Variable
from torch.utils.data import DataLoader, Dataset
import torch.nn.functional as F
from torchvision import transforms as T

def read_data():
    """
    read img_file data
    :return: img_path, img_name
    """
    train_data = glob.glob('../train-images/*.jpeg')
    train_label = np.array(
        [train_data[index].split('/')[-1].split('.')[0].split('_')[0] for index in
         range(len(train_data))])
    return train_data, train_label

out_place = (
    "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q',
    'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z')
transform = T.Compose([
   # T.Resize((128,128)),
    T.ToTensor(),
    T.Normalize(std=[0.5, 0.5, 0.5], mean=[0.5, 0.5, 0.5])
])

def one_hot(word: str) -> np.array:
    tmp = []
    # for i in range(len(word)):
    for i in range(4):
        item_tmp = [0 for x in range(144)]   # [0 for x in range(len(word) * len(out_place))]
        word_idx = out_place.index(word[i].lower())
        item_tmp[i*36+word_idx] = 1
        tmp.append(item_tmp)
    return np.array(tmp)

class DataSet(Dataset):
    def __init__(self):
        self.img_path, self.label = read_data()

    def __getitem__(self, index):
        img_path = self.img_path[index]
        img = Image.open(img_path).convert("RGB")
        img = transform(img)
        label = torch.from_numpy(one_hot(self.label[index])).float()
        return img, label

    def __len__(self):
        return len(self.img_path)

data = DataSet()
data_loader = DataLoader(data, shuffle=True, batch_size=64, drop_last=True)

class CNN_Network(nn.Module):
    def __init__(self):
        super(CNN_Network, self).__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 16, stride=1, kernel_size=3, padding=1),
            nn.BatchNorm2d(16),
            nn.ReLU(inplace=True)

        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, stride=1, kernel_size=3, padding=1),
            nn.BatchNorm2d(32),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(stride=2, kernel_size=2),  # 30 80
        )
        self.layer3 = nn.Sequential(
            nn.Conv2d(32, 64, stride=1, kernel_size=3, padding=1),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2, stride=2),   # 15 40
        )

        self.fc = nn.Sequential(
            nn.Linear(256 * 15 * 40, 2048),
            nn.ReLU(inplace=True),
            nn.Linear(2048, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, 40)
        )

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)

        x = x.view(x.size(0), -1)
        x = self.fc(x)
        x = F.softmax(x, dim=-1)
        return x

model = CNN_Network()

model.train()
optimizer = torch.optim.Adam(model.parameters(), lr=0.002)

error = nn.MultiLabelSoftMarginLoss()

for i in range(5):
    for (batch_x, batch_y) in data_loader:
        optimizer.zero_grad()
        image = Variable(batch_x)
        label = Variable(batch_y)

        out = model(image)

        loss = error(out, label)

        print(loss)

        loss.backward()
        optimizer.step()

torch.save(model.state_dict(), "model.pth")

my image size [70, 26]  and one of images name is  okzi.jpeg
I am first touch pytorch so i don`t understand this question
PP7{}5GRRC7EVLMTQJYD1L926×537 77.6 KB

Summary
This text will be hidden

Context:
The activation input x to self.fc doesn’t have the expected number of features, so you would need to change the in_features of the first nn.Linear layer in self.fc to 13056.
Also, unrelated to this issue, but Variables are deprecated since PyTorch 0.4.0, so you can use tensors now.


ValueError: too many values to unpack (expected 3),

Question:
Hey,
I want to predict multiple objects in images and also their bounding boxes.
So at first I load the data from xml file and because of the different numbers of values for labels and boxes in each image I use an own collate-function.
The problem is that I get these error message: ValueError: too many values to unpack (expected 3).
And here the full messages:
File "...", line 169, in <module>     main()   File "...", line 46, in main        for batch, data in enumerate(train_data):   File "...PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\torch\utils\data\dataloader.py", line 521, in __next__     data = self._next_data()   File "...PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data     data = self._dataset_fetcher.fetch(index)  # may raise StopIteration   File "...PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\LocalCache\local-packages\Python37\site-packages\torch\utils\data\_utils\fetch.py", line 52, in fetch     return self.collate_fn(data)   File "...", line 12, in my_collate     img,b,labels = zip(*batch)
I am just a beginner in deep learning, so it would be great if someone could help.
Here is some of my written code:
# libraries
import os
import numpy as np
import cv2
import matplotlib.pyplot as plt
from matplotlib.patches import Rectangle
import torch
from xml.etree import ElementTree

import config
def my_collate(batch):
    img,b,labels = zip(*batch)
    img = torch.LongTensor([config.ws.transform(i,max_len=config.max_len) for i in img])
    labels = torch.LongTensor(labels)
    b = torch.LongTensor(b)
    return img, b,labels

def main():
    print("load data ...")
    class_names = ['person','person-like']
    class_names_label = {class_name:i for i, class_name in enumerate(class_names)}
    size = (256,256)
    batch_size=32

    (train_images, b_tr, b_label_tr),(test_images,b_te,b_label_te),(val_images, b_v,b_label_v) = load_data(class_names_label, size)
    train_dataset=(train_images, b_tr, b_label_tr)
    test_dataset=(test_images, b_te,b_label_te)
    val_dataset=(val_images, b_v, b_label_v)
    train_data=torch.utils.data.DataLoader(train_dataset, batch_size, shuffle=True, collate_fn=my_collate)
    test_data=torch.utils.data.DataLoader(test_dataset, batch_size, shuffle=True,collate_fn=my_collate)
    val_data=torch.utils.data.DataLoader(val_dataset, batch_size, shuffle=True,collate_fn=my_collate)

    #show example images
    i=np.random.randint(0,train_images.shape[0]-1)
    #print("i: ", i, " , ", b_tr[i])
    #print(b_label_tr[i])
    plot_img_withRectangle(i=i, boxes=b_tr,images=train_images,box_labels=b_label_tr,img_size=size)
    #plt.show()
    plt.close()

    #train: train_data
    lr=0.2
    num_epochs=2
    for epoch in range(num_epochs):
        print()
        for batch, data in enumerate(train_data):
            i=0     #irrelevant value

#functions
def load_data(class_names_label, size):
    datasets = ['Train/Train', 'Test/Test', 'Val/Val']
    output=[]
    for dataset in datasets:
        imags = []
        boxes_all=[]
        label_boxes=[]
        directoryA = "PedestrianDetection/" + dataset +"/Annotations/"
        directoryIMG = "PedestrianDetection/" + dataset +"/JPEGImages/"
        file = os.listdir(directoryA)
        img = os.listdir(directoryIMG)
        file.sort()
        img.sort()
        i = 0
        for xml in file:
            xmlf = os.path.join(directoryA,xml)
            root = ElementTree.parse(xmlf)
            vb = root.findall('object')

            s=root.findall('size')
            w=float(s[0].find('width').text)
            h=float(s[0].find('height').text)
            #boxes
            boxes_img=[]
            idx=0
            label_boxes_img=[]
            for anz in vb:
                #label from box in image
                label_box=vb[idx].find('name').text
                label_boxes_img.append(class_names_label[label_box])
                #box
                single_box=[]
                xmin=float(vb[idx].find('bndbox/xmin').text)
                ymin=float(vb[idx].find('bndbox/ymin').text)
                xmax=float(vb[idx].find('bndbox/xmax').text)
                ymax=float(vb[idx].find('bndbox/ymax').text)
                #normalize box
                if h>0 and w>0:
                    x_min=xmin*(size[1]/w)
                    y_min=ymin*(size[0]/h)
                    x_max=xmax*(size[1]/w)
                    y_max=ymax*(size[0]/h)
                single_box=[(x_min), (y_min), (x_max), (y_max)]
                boxes_img.append(single_box)
                idx+=1
            boxes_all.append(boxes_img)
            label_boxes.append(label_boxes_img)
            #image
            img_path = directoryIMG + img[i]
            curr_img = cv2.imread(img_path)
            curr_img = cv2.resize(curr_img, size)
            imags.append(curr_img)
            i +=1
        imags = np.array(imags, dtype='float32')
        imags = imags / 255

        output.append((imags, boxes_all, label_boxes))
    return output
def plot_img_withRectangle(i,boxes,images,box_labels,img_size):
    plt.figure(1)
    plt.title(box_labels[i])
    img_cv=cv2.cvtColor(images[i], cv2.COLOR_BGR2RGB)
    plt.imshow(img_cv)
    line_width=1.5
    xmin_n_img=[]
    ymin_n_img=[]
    xmax_n_img=[]
    ymax_n_img=[]
    width_n_img=[]
    height_n_img=[]
    ytop_img=[]
    zz=0
    for anz in range(len(boxes[i])):
        xmin_n=(boxes[i][zz][0])    #[image][single box][idx]
        ymin_n=(boxes[i][zz][1])
        xmax_n=(boxes[i][zz][2])
        ymax_n=(boxes[i][zz][3])
        width_n=(xmax_n-xmin_n)
        height_n=(ymax_n-ymin_n)
        ytop=ymin_n
        if (height_n+ytop) > (img_size[0]-line_width):
            height_n=height_n-line_width
        if (width_n+xmin_n) > (img_size[1]-line_width):
            width_n=width_n-line_width
        xmin_n_img.append(xmin_n)
        ymin_n_img.append(ymin_n)
        xmax_n_img.append(xmax_n)
        ymax_n_img.append(ymax_n)
        width_n_img.append(width_n)
        height_n_img.append(height_n)
        ytop_img.append(ytop)
        #box_label
        if box_labels[i][zz]==0:    #person
            plt.text(xmin_n,ytop, 'person', color='yellow')
        elif box_labels[i][zz]==1:  #person-like
            plt.text(xmin_n,ytop, 'person-like', color='blue')
        zz+=1
    z=0
    for anz in range(len(boxes[i])):
        if box_labels[i][z]==0:
            plt.gca().add_patch(Rectangle(
                (xmin_n_img[z],ytop_img[z]),width_n_img[z],height_n_img[z],
                edgecolor='yellow',
                facecolor='none',
                lw=line_width
            ))
        elif box_labels[i][z]==1:
            plt.gca().add_patch(Rectangle(
                (xmin_n_img[z],ytop_img[z]),width_n_img[z],height_n_img[z],
                edgecolor='blue',
                facecolor='none',
                lw=line_width
            ))
        z+=1

if __name__=="__main__":
    main()

Context:
The error is raised in:
img,b,labels = zip(*batch)

so check what batch is, how many elements it contains, and how to unwrap it.


Question: Is there a way to train faster on a single gpu, I have a 16 core cpu, and it seems like increasing the number of workers on the dataloader slows down training? Can I get any hints in this regards. I’m somewhat new in this domain. Thank you.

Context:
This post gives a great overview how to handle data loading bottlenecks.
In particular this section might be interesting for you:

Beyond an optimal number (experiment!), throwing more worker processes at the IOPS barrier WILL NOT HELP, it’ll make it worse. You’ll have more processes trying to read files at the same time, and you’ll be increasing the shared memory consumption by significant amounts for additional queuing, thus increasing the paging load on the system and possibly taking you into thrashing territory that the system may never recover from


Capture CUDAGraph without using double the memory

Question:
Hi!
I can successfully capture the CUDAGraph and replay. I took the API example from this blog and modified it for my own model. Basically, I can forward and backward run my model normally with a certain batch size (25). But when I need to capture the graph, the reserved memory suddenly doubles, making me only able to do batch size 8.
...
device = 'cpu'
if torch.cuda.is_available():
    device = 'cuda'

train_x, train_y = data_generator(T, seq_len, n_train)
model = TCN(1, n_classes, channel_sizes, kernel_size, dropout=dropout).to(device)
criterion = torch.nn.CrossEntropyLoss().to(device)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

print('Before stream s, allocated: ', torch.cuda.memory_allocated('cuda'), '; reserved: ', torch.cuda.memory_reserved('cuda'))
# warmup
s = torch.cuda.Stream()
s.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(s):
    for i in range(3):
        inputs = train_x[batch_size*i:batch_size*(i+1)].unsqueeze(1).contiguous().to(device)
        labels = train_y[batch_size*i:batch_size*(i+1)].to(device)
        optimizer.zero_grad(set_to_none=True)
        y_pred = model(inputs)
        loss = criterion(y_pred.view(-1, n_classes), labels.view(-1))
        loss.backward()
        optimizer.step()
torch.cuda.current_stream().wait_stream(s)

print('After stream s before graph g, allocated: ', torch.cuda.memory_allocated('cuda'), '; reserved: ', torch.cuda.memory_reserved('cuda'))
print('Start capturing CUDA graph.')

# capture
g = torch.cuda.CUDAGraph()
# Sets grads to None before capture, so backward() will create
# .grad attributes with allocations from the graph's private pool
optimizer.zero_grad(set_to_none=True)
static_inputs = train_x[:batch_size].unsqueeze(1).contiguous().to(device)
static_labels = train_y[:batch_size].to(device)
with torch.cuda.graph(g):
    static_y_pred = model(static_inputs)
    static_loss = criterion(static_y_pred.view(-1, n_classes), static_labels.view(-1))
    static_loss.backward()
    optimizer.step()

print('After graph g before replay, allocated: ', torch.cuda.memory_allocated('cuda'), '; reserved: ', torch.cuda.memory_reserved('cuda'))

for i in range(30):
    inputs = train_x[batch_size*i:batch_size*(i+1)].unsqueeze(1).contiguous().to(device)
    labels = train_y[batch_size*i:batch_size*(i+1)].to(device)
    static_inputs.copy_(inputs)
    static_labels.copy_(labels)
    g.replay()

print('After replay, allocated: ', torch.cuda.memory_allocated('cuda'), '; reserved: ', torch.cuda.memory_reserved('cuda'))

The output is:
Before stream s, allocated:  2978816 ; reserved:  4194304
After stream s before graph g, allocated:  13397522944 ; reserved:  15101591552
Start capturing CUDA graph.
After graph g before replay, allocated:  13400187392 ; reserved:  30045896704
After replay, allocated:  13400187392 ; reserved:  30045896704

If I try a batch size larger (9, 25, etc), it works up to capturing the graph. The error output is:
Traceback (most recent call last):
  File "/gpfs/fs1/home/minzhao.liu/TCN/cuda_graph.py", line 123, in <module>
    optimizer.step()
  File "/home/minzhao.liu/.conda/envs/qtensor-torch/lib/python3.9/site-packages/torch/cuda/graphs.py", line 149, in __exit__
    self.cuda_graph.capture_end()
  File "/home/minzhao.liu/.conda/envs/qtensor-torch/lib/python3.9/site-packages/torch/cuda/graphs.py", line 71, in capture_end
    super(CUDAGraph, self).capture_end()
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

Also, can I save the graph and run it on a different machine? If anyone can recommend some resources and more examples of CUDA graphs with PyTorch, I would really appreciate since I have only found a few minimal examples.

Context:
What jumps out at me is
Before stream s, allocated:  2978816...
After stream s before graph g, allocated:  13397522944

The model evidently isn’t that big (about 3 MB) but somehow after warmup the allocated memory shoots up to 13 GB.
I’m not surprised the reserved memory gets pushed up to ~15 GB (it increases to accommodate the high-water mark of the built graph). But after backward(), the graph should be torn down, and the allocated memory should settle back to whatever the model params and optimizer states need which should be several times the model’s param set size, in other words 3 MB x N where N is a small integer.
Can you add
gc.collect()
torch.cuda.empty_cache()

just above
print('After stream s before graph g, allocated: ', torch.cuda.memory_allocated('cuda'), '; reserved: ', torch.cuda.memory_reserved('cuda'))

and report the values?

can I save the graph and run it on a different machine

No, there’s currently no way to serialize and deserialize graphs.

more examples of CUDA graphs with PyTorch

The master docs are the best resource.
https://pytorch.org/docs/master/notes/cuda.html#cuda-graphs
Some internal people are adding cuda graphs to DeepLearningExamples/PyTorch at master · NVIDIA/DeepLearningExamples · GitHub 1, but I don’t know how far that effort has progressed.


How can I extract features from all the layer from loaded GNN model?

Question:
Recently, I git cloned a GNN network and am trying to get the output from all the layers (as I am not sure what each layer are named).
There is a file in the GitHub repo that does the prediction by loading a pre-trained model with an example input i.e. pretrained.py and I am trying to add some additional code in it to extract outputs.
I have tried adding the following code just after the model.eval() in the above-mentioned file that was mentioned in other threads (which was used to extract intermediate layer output from loaded CNN models):
activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook

for name, layer in model.named_modules():
    layer.register_forward_hook(get_activation(name))

But I am getting the following error:
Traceback (most recent call last):
  File "alignn/pretrained.py", line 248, in <module>
    out_data = get_prediction(
  File "alignn/pretrained.py", line 221, in get_prediction
    model([g.to(device), lg.to(device)])
  File "/home/vgf3011/.virtualenvs/alignn/lib64/python3.8/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/home/vgf3011/vishu/alignn/alignn/models/alignn.py", line 290, in forward
    x, y, z = alignn_layer(g, lg, x, y, z)
  File "/home/vgf3011/.virtualenvs/alignn/lib64/python3.8/site-packages/torch/nn/modules/module.py", line 1120, in _call_impl
    result = forward_call(*input, **kwargs)
  File "/home/vgf3011/vishu/alignn/alignn/models/alignn.py", line 163, in forward
    x, m = self.node_update(g, x, y)
  File "/home/vgf3011/.virtualenvs/alignn/lib64/python3.8/site-packages/torch/nn/modules/module.py", line 1123, in _call_impl
    hook_result = hook(self, input, result)
  File "alignn/pretrained.py", line 168, in hook
    activation[name] = output.detach()
AttributeError: 'tuple' object has no attribute 'detach'

Is there anything that I am doing wrong or do I have to be more specific on things to extract output from each of the layers?

Context: As given in the error message output is a tuple, so you can’t call detach() on it (it’s a tensor operator) and would need to unwrap the tuple first.


Question:
When building from the source it throws up nvlink error. I have followed these instructions 24.
[386/3538] Performing build step for 'nccl_external'
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a
cd /codehub/external/pytorch/third_party/nccl/nccl && env CCACHE_DISABLE=1 SCCACHE_DISABLE=1 make CXX=/usr/bin/c++ CUDA_HOME=/usr/local/cuda NVCC=/usr/local/cuda/bin/nvcc "NVCC_GENCODE=-gencode=arch=compute_35,code=sm_35 -gencode=arch=compute_37,code=sm_37 -gencode=arch=compute_50,code=sm_50 -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_53,code=sm_53 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75" BUILDDIR=/codehub/external/pytorch/build/nccl VERBOSE=0 -j && /root/anaconda3/bin/cmake -E touch /codehub/external/pytorch/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build
make -C src build BUILDDIR=/codehub/external/pytorch/build/nccl
make[1]: Entering directory '/codehub/external/pytorch/third_party/nccl/nccl/src'
Generating nccl.h.in                           > /codehub/external/pytorch/build/nccl/include/nccl.h
Grabbing   include/nccl_net.h                  > /codehub/external/pytorch/build/nccl/include/nccl_net.h
Compiling  init.cc                             > /codehub/external/pytorch/build/nccl/obj/init.o
Compiling  channel.cc                          > /codehub/external/pytorch/build/nccl/obj/channel.o
Compiling  bootstrap.cc                        > /codehub/external/pytorch/build/nccl/obj/bootstrap.o
Compiling  transport.cc                        > /codehub/external/pytorch/build/nccl/obj/transport.o
Compiling  enqueue.cc                          > /codehub/external/pytorch/build/nccl/obj/enqueue.o
Compiling  misc/group.cc                       > /codehub/external/pytorch/build/nccl/obj/misc/group.o
Compiling  misc/nvmlwrap.cc                    > /codehub/external/pytorch/build/nccl/obj/misc/nvmlwrap.o
Compiling  misc/ibvwrap.cc                     > /codehub/external/pytorch/build/nccl/obj/misc/ibvwrap.o
Compiling  misc/rings.cc                       > /codehub/external/pytorch/build/nccl/obj/misc/rings.o
Compiling  misc/utils.cc                       > /codehub/external/pytorch/build/nccl/obj/misc/utils.o
Compiling  misc/argcheck.cc                    > /codehub/external/pytorch/build/nccl/obj/misc/argcheck.o
Compiling  misc/trees.cc                       > /codehub/external/pytorch/build/nccl/obj/misc/trees.o
Compiling  misc/topo.cc                        > /codehub/external/pytorch/build/nccl/obj/misc/topo.o
Compiling  transport/p2p.cc                    > /codehub/external/pytorch/build/nccl/obj/transport/p2p.o
Compiling  transport/shm.cc                    > /codehub/external/pytorch/build/nccl/obj/transport/shm.o
Compiling  transport/net.cc                    > /codehub/external/pytorch/build/nccl/obj/transport/net.o
Compiling  transport/net_socket.cc             > /codehub/external/pytorch/build/nccl/obj/transport/net_socket.o
In file included from bootstrap.cc:12:
include/socket.h: In function ‘ncclResult_t connectAddress(int*, socketAddress*)’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from bootstrap.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compiling  transport/net_ib.cc                 > /codehub/external/pytorch/build/nccl/obj/transport/net_ib.o
Compiling  collectives/all_reduce.cc           > /codehub/external/pytorch/build/nccl/obj/collectives/all_reduce.o
In file included from bootstrap.cc:12:
include/socket.h: In function ‘int findInterfaceMatchSubnet(char*, socketAddress*, socketAddress, int, int)’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from bootstrap.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compiling  collectives/all_gather.cc           > /codehub/external/pytorch/build/nccl/obj/collectives/all_gather.o
Compiling  collectives/broadcast.cc            > /codehub/external/pytorch/build/nccl/obj/collectives/broadcast.o
Compiling  collectives/reduce.cc               > /codehub/external/pytorch/build/nccl/obj/collectives/reduce.o
Compiling  collectives/reduce_scatter.cc       > /codehub/external/pytorch/build/nccl/obj/collectives/reduce_scatter.o
make[2]: Entering directory '/codehub/external/pytorch/third_party/nccl/nccl/src/collectives/device'
Generating nccl.pc.in                          > /codehub/external/pytorch/build/nccl/lib/pkgconfig/nccl.pc
Generating rules                               > /codehub/external/pytorch/build/nccl/obj/collectives/device/Makefile.rules
In file included from bootstrap.cc:12:
include/socket.h: In function ‘ncclResult_t bootstrapNetInit()’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from bootstrap.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bootstrap.cc:44:58: warning: ‘%s’ directive output may be truncated writing up to 1023 bytes into a region of size between 1017 and 1018 [-Wformat-truncation=]
           snprintf(line+strlen(line), 1023-strlen(line), " [%d]%s:%s", i, bootstrapNetIfNames+i*MAX_IF_NAME_SIZE,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
               socketToString(&bootstrapNetIfAddrs[i].sa, addrline));
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from bootstrap.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64:35: note: ‘__builtin___snprintf_chk’ output 6 or more bytes (assuming 1030) into a destination of size 1023
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from transport/net_socket.cc:9:
include/socket.h: In function ‘ncclResult_t connectAddress(int*, socketAddress*)’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_socket.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from transport/net_ib.cc:9:
include/socket.h: In function ‘ncclResult_t ncclIbInit(ncclDebugLogger_t)’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_ib.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from transport/net_ib.cc:9:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_ib.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from transport/net_socket.cc:9:
include/socket.h: In function ‘ncclResult_t ncclSocketInit(ncclDebugLogger_t)’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_socket.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from transport/net_socket.cc:9:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_socket.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
transport/net_socket.cc:40:58: warning: ‘%s’ directive output may be truncated writing up to 1023 bytes into a region of size between 1017 and 1018 [-Wformat-truncation=]
           snprintf(line+strlen(line), 1023-strlen(line), " [%d]%s:%s", i, ncclNetIfNames+i*MAX_IF_NAME_SIZE,
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
               socketToString(&ncclNetIfAddrs[i].sa, addrline));
               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_socket.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:64:35: note: ‘__builtin___snprintf_chk’ output 6 or more bytes (assuming 1030) into a destination of size 1023
   return __builtin___snprintf_chk (__s, __n, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        __bos (__s), __fmt, __va_arg_pack ());
        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from transport/net_ib.cc:9:
include/socket.h: In function ‘ncclResult_t ncclIbConnect(int, void*, void**)’:
include/socket.h:41:16: warning: ‘<’ directive writing 1 byte into a region of size between 0 and 1024 [-Wformat-overflow=]
   sprintf(buf, "%s<%s>", host, service);
                ^~~~~~~~
In file included from /usr/include/stdio.h:862,
                 from include/debug.h:11,
                 from include/core.h:13,
                 from transport/net_ib.cc:8:
/usr/include/x86_64-linux-gnu/bits/stdio2.h:33:34: note: ‘__builtin___sprintf_chk’ output between 3 and 1058 bytes into a destination of size 1024
   return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1,
          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
       __bos (__s), __fmt, __va_arg_pack ());
       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_i8.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_u8.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_i32.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_u32.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_i64.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_u64.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_f16.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_f32.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_sum_f64.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_i8.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_u8.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_i32.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_u32.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_i64.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_u64.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_f16.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_f32.o
Compiling  all_gather.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_gather_prod_f64.o
.
.
.
.
.
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_min_f16.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_min_f32.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_min_f64.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_i8.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_u8.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_i32.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_u32.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_i64.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_u64.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_f16.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_f32.o
Compiling  all_reduce.cu                       > /codehub/external/pytorch/build/nccl/obj/collectives/device/all_reduce_max_f64.o
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_f648ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_f648ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_f328ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_f328ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_f168ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_f168ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_u648ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_u648ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_i648ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_i648ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_u328ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_u328ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterTreeLLKernel_sum_i328ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z37ncclReduceScatterRingLLKernel_sum_i328ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z36ncclReduceScatterTreeLLKernel_sum_u88ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z36ncclReduceScatterRingLLKernel_sum_u88ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z36ncclReduceScatterTreeLLKernel_sum_i88ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z36ncclReduceScatterRingLLKernel_sum_i88ncclColl' with max regcount of 80 calls function '_Z24ncclAllReduceRing_sum_i8P14CollectiveArgs' with regcount of 96 (target: sm_53)
.
.
.
.
.
some 10000 lines of nvlink error
.
.
.
.
.
.
nvlink error   : entry function '_Z33ncclAllReduceTreeLLKernel_sum_u328ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z33ncclAllReduceRingLLKernel_sum_u328ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z33ncclAllReduceTreeLLKernel_sum_i328ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z33ncclAllReduceRingLLKernel_sum_i328ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z32ncclAllReduceTreeLLKernel_sum_u88ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z32ncclAllReduceRingLLKernel_sum_u88ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z32ncclAllReduceTreeLLKernel_sum_i88ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
nvlink error   : entry function '_Z32ncclAllReduceRingLLKernel_sum_i88ncclColl' with max regcount of 80 calls function '_Z29ncclReduceScatterRing_max_f64P14CollectiveArgs' with regcount of 96 (target: sm_53)
Makefile:68: recipe for target '/codehub/external/pytorch/build/nccl/obj/collectives/device/devlink.o' failed
make[2]: *** [/codehub/external/pytorch/build/nccl/obj/collectives/device/devlink.o] Error 255
make[2]: Leaving directory '/codehub/external/pytorch/third_party/nccl/nccl/src/collectives/device'
Makefile:49: recipe for target '/codehub/external/pytorch/build/nccl/obj/collectives/device/colldevice.a' failed
make[1]: *** [/codehub/external/pytorch/build/nccl/obj/collectives/device/colldevice.a] Error 2
make[1]: Leaving directory '/codehub/external/pytorch/third_party/nccl/nccl/src'
Makefile:25: recipe for target 'src.build' failed
make: *** [src.build] Error 2
[387/3538] Generating src/x86_64-fma/blas/shdotxf.py.o
[388/3538] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_generic.dir/src/EmbeddingSpMDM.cc.o
[389/3538] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_generic.dir/src/EmbeddingSpMDMNBit.cc.o
[390/3538] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8Depthwise3DAvx2.cc.o
[391/3538] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwisePerChannelQuantAvx2.cc.o
[392/3538] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8Depthwise3x3Avx2.cc.o
[393/3538] Building CXX object third_party/fbgemm/CMakeFiles/fbgemm_avx2.dir/src/FbgemmI8DepthwiseAvx2.cc.o
ninja: build stopped: subcommand failed.
Building wheel torch-1.5.0a0+9857d9b
-- Building version 1.5.0a0+9857d9b
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/codehub/external/pytorch/torch -DCMAKE_PREFIX_PATH=/root/anaconda3 -DNUMPY_INCLUDE_DIR=/root/anaconda3/lib/python3.7/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/root/anaconda3/bin/python -DPYTHON_INCLUDE_DIR=/root/anaconda3/include/python3.7m -DPYTHON_LIBRARY=/root/anaconda3/lib/libpython3.7m.so.1.0 -DTORCH_BUILD_VERSION=1.5.0a0+9857d9b -DUSE_NUMPY=True /codehub/external/pytorch
cmake --build . --target install --config Release -- -j 8
Traceback (most recent call last):
  File "setup.py", line 737, in <module>
    build_deps()
  File "setup.py", line 316, in build_deps
    cmake=cmake)
  File "/codehub/external/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/codehub/external/pytorch/tools/setup_helpers/cmake.py", line 341, in build
    self.run(build_args, my_env)
  File "/codehub/external/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/root/anaconda3/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '8']' returned non-zero exit status 1.

Context:
That’s strange. Somehow -gencode=arch=compute_53,code=sm_53 is generated.
Could you try to build it via:
TORCH_CUDA_ARCH_LIST="5.0" python setup.py install


Torch.index_put on GPU with accumulate=True

Question:
I’ve seen two posts about index_put on GPU: here and here. They are both marked as solved, but it’s not working for me. It’s fine on the CPU, and on GPU with accumulate=False, but when accumulate=True I get
RuntimeError: linearIndex.numel()*sliceSize*nElemBefore == value.numel()INTERNAL ASSERT FAILED at "/opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/cuda/Indexing.cu":250, please report a bug to PyTorch. number of flattened indices did not match number of elements in the value tensor251

Steps to reproduce:
t1 = torch.zeros(10).long().cuda()
t2 = torch.randint(10, size=(25,)).cuda()
t1.index_put(indices=[t2], values=torch.tensor(1), accumulate=True)

Pytorch version: 1.10.1

Context: Could you update to the latest nightly as I cannot reproduce the issue, please?


Looking for insights on debugging issues loading state dict of a net trained on GPU on CPU

Question:
Hi All,
I’m at a loss for the time being on how to proceed with debugging problematic loading of a GPU-trained model on CPU. The model is a generative flow that I sample time series information from. I can provide more detail as necessary, but starting more simply for now, here’s the steps of what I’m doing as I go by the documentation for loading a GPU net on CPU.
device = torch.device('cpu')
net = SDEFlow(device, obs_model, state_dim_SCON, t, dt_flow, n, num_layers = 5) #instantiates model, leaving arguments for future explanation if necessary
net.load_state_dict(torch.load(net_state_dict_save_string), map_location = 'cpu')

net.eval()
x, _ = net(batch_size) #Forward method of net outputs two things, and x is the matrix time series.

#various plotting code from here

and then I end up with a problematic figure that doesn’t align with my GPU results:

image900×864 241 KB

Re-loading the model on the device on which the training occurred
device = torch.device('cuda')
net = SDEFlow(device, obs_model, state_dim_SCON, t, dt_flow, n, num_layers = 5)
net.load_state_dict(torch.load(net_state_dict_save_string), map_location = 'cuda')

net.eval()
x, _ = net(batch_size)

#various plotting code from here.

I’m able to get back the results that I expect:

net_iter_105000_warmup_1000_t_5000_dt_1.0_batch_31_samples_31_layers_5_lr_0.01_warmup_lr_1e-06_sd_scale_0.25_SCON-C_CO2_logit_short_2022_01_21_05_31_02_test21920×1920 197 KB

I don’t think the bug is in my plotting code, since I can already tell things are wrong via print output of x on the CPU, so I didn’t include my plotting code for now either to avoid extraneous information.
Am I missing something obviously wrong with respect to the order that I’m doing things? I made sure that the flow model being instantiated was the exact same, so it appears that something is not being converted correctly in the saving of the state dict? In the training, I’m saving the net state dict along the lines of torch.save(net.state_dict(), net_state_dict_file_string). Evaluation batch_size is equivalent on both GPU and CPU.
Happy to provide more info. I appreciate your time and patience in reading through this question.

Context:
Do I understand the issue correctly that your model is performing as expected if you are training and restoring the model on the GPU, but fails if you are training on the GPU and restoring on the CPU?
If so, which PyTorch version are you using? And if not the latest one, could you update it to the latest stable or nightly release?
A while ago (maybe ~6 months) I’ve debugged an issue where the compression during torch.save created invalid data when e.g. nn.Embedding layers were stored from the GPU (it’s already fixed). If you could resave the model, could you disable the compression via torch.save(..., _use_new_zipfile_serialization=False) and check if reloading this state_dict would work?


Difference in type between Input type and Weight type with ModuleList

Question:
Hello guys,
I have a problem with a list of modules. I have coded a module called BranchRoutingModule. I would like to create a list from this module.
I have the following code:
def _branch_routings(self):
    # structure = [nn.ModuleList([BranchRoutingModule(in_channels=self.in_channels) for j in range(int(pow(2, i)))]) for i in range(self.tree_height - 1)] # [[None], [None, None]] for tree height = 3
    structure = [[None for j in range(int(pow(2, i)))] for i in range(self.tree_height - 1)] # [[None], [None, None]] for tree height = 3

    cur = 0
    for i in range(self.tree_height - 1):
      for j in range(int(pow(2, i))):
        self.__setattr__('branch_routing_module' + str(cur), BranchRoutingModule(in_channels=self.in_channels))
        structure[i][j] = self.__getattr__('branch_routing_module' + str(cur))
        cur += 1
    return structure

I have first tried using nn.ModuleList (commented out at the top) but I get the following error: “Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same”
However if I use setattr and getattr, I get no errors and my model works fine.
Why is that? I don’t understand why setattr and getattr fix the problem.
I am using CUDA.
Thank you and regards,
Antoine

Context:
It seems you were trying to register a plain Python list containing nn.ModuleLists in your initial approach, which won’t work.
Make sure to register nn.Modules directly or nn.ModuleLists (not plain Python lists).


Question:
Hi,
I’m currently working on a single-GPU system with limited GPU memory where multiple torch models are offered as “services” that run in separate python processes.
Ideally, I would like to be able to free the GPU memory for each model on demand without killing their respective python process. However, if I only delete the models (and empty the cache) without killing the process, I’m still stuck with the memory usage of the cuda context, which is about 1.2 GB per process in my case.
In general, the memory usage of the context seems extremely high to me, especially when considering that the model itself only adds about 200MB to it while it’s loaded.
So my question is: Are there any options to decrease the memory footprint of the cuda context or to get rid of it altogether when running inference on the models?
I’d be grateful for advice!

Context:
alinak:

So my question is: Are there any options to decrease the memory footprint of the cuda context or to get rid of it altogether when running inference on the models?

You could reduce the CUDA context size by removing kernels e.g. via dropping libraries such as MAGMA or cuDNN, if these are not used.
Besides that I’m not aware of reducing the operator set in the framework itself.


Problem calculating accuracy on multiclass classfication

Question:
Hi, I am facing a problem when I try to compute accuracy on a multiclass classification problem with the Accuracy() function from torchmetrics
Here is the training function I created:
from torchmetrics import Accuracy
accuracy = Accuracy()

def training(epoch, model, train_loader, optimizer, criterion):
  "Training over an epoch"
  metric_monitor = MetricMonitor()

  model.train()

  for batch in train_loader:
    images = batch['t1'][tio.DATA].cuda()
    labels = batch['label'].cuda()
    output = F.softmax(model(images), dim=0)

    loss = criterion(output, labels)

    output = output.data.max(dim=1,keepdim=True)[1]

    acc = accuracy(output, labels)

    metric_monitor.update("Loss", loss.item())
    metric_monitor.update("Accuracy", acc)

    optimizer.zero_grad()

    loss.backward()
    optimizer.step()
  print("[Epoch: {epoch:03d}] Train      | {metric_monitor}".format(epoch=epoch, metric_monitor=metric_monitor))
  return metric_monitor.metrics['Loss']['avg'], metric_monitor.metrics['Accuracy']['avg']

When I try to call this function to train the model I get the following error :
ValueError: If preds have one dimension more than target, preds should be a float tensor.
Does anybody know what I am doing wrong ?

Context:
@gkrisp9 This is a classic case of CrossEntropy evaluation
https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html
When you have multi label classification, your output dimension becomes
batch_size, num_of_classes
And your input target is batch_size
The solution would be to add this extra line
output = output.data.max(dim=1,keepdim=True)[1]
output = output.view(-1)
acc = accuracy(output, labels)


Dataloader issues with multiprocessing when i do torch.multiprocessing.set_start_method(“spawn”, force = True)

Question:
Does anyone know what this error is?
 Traceback (most recent call last):
 File "./train.py", line 506, in <module>
e = t.train()
File "./train.py", line 188, in train
self.train_generator = iter(self.train_loader)
File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 278, in __iter__
return _MultiProcessingDataLoaderIter(self)
File "/usr/local/lib/python3.5/dist-packages/torch/utils/data/dataloader.py", line 682, in __init__
w.start()
File "/usr/lib/python3.5/multiprocessing/process.py", line 105, in start
self._popen = self._Popen(self)
File "/usr/lib/python3.5/multiprocessing/context.py", line 212, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/usr/lib/python3.5/multiprocessing/context.py", line 274, in _Popen
return Popen(process_obj)
File "/usr/lib/python3.5/multiprocessing/popen_spawn_posix.py", line 33, in __init__
super().__init__(process_obj)
File "/usr/lib/python3.5/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/usr/lib/python3.5/multiprocessing/popen_spawn_posix.py", line 48, in _launch
reduction.dump(process_obj, fp)
File "/usr/lib/python3.5/multiprocessing/reduction.py", line 59, in dump
ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'Environment'>: attribute lookup Environment on builtins failed

My dataloader code is as follows:
from torch.utils.data import Dataset, DataLoader
import torch
from torchvision import transforms
import pandas as pd
import numpy as np
from torch.utils.data.sampler import SubsetRandomSampler
import os
import lmdb
import pyarrow
import lz4framed
from typing import Any
from PIL import Image
from lmdbconverter import convert

class InvalidFileException(Exception):
pass

class BirdDatesetLMDB(Dataset):
def __init__(self, train = True, store_path = 'CUB_200_2011', transform=None):
    super().__init__()

    tr_te_fn = os.path.join(store_path, 'train_test_split.txt')
    labels = dict([map(int, line.rstrip('\n').split(' ')) for line in open(os.path.join(store_path, 'image_class_labels.txt'))])
    df = pd.read_csv(os.path.join(store_path, 'attributes/images_by_attributes.csv'), index_col=0)
    struct_features = dict(zip(range(1,df.shape[0]+1), df.values))

    bdng_box = dict(
        map(lambda x: (x[0], x[1:]),
            [list(map(int, map(float,
                line.rstrip('\n').split(' '))))
            for line in open(os.path.join(store_path, 'bounding_boxes.txt'))]))

    self.labels = []
    self.struct = []
    self.bdng_box = []

    tr_te = np.loadtxt(tr_te_fn)

    if train:
        lmdb_store_path = store_path + '/train.db'
        ids = np.where(tr_te[:,1] == 1)[0] + 1
    else:
        lmdb_store_path = store_path + '/test.db'
        ids = np.where(tr_te[:,1] == 0)[0] + 1

    self.N=len(ids)

    for i in range(self.N):
        img_id = ids[i]
        self.labels.append(labels[img_id]-1)
        self.struct.append(struct_features[img_id])

        if bdng_box is not None:
            self.bdng_box.append(bdng_box[img_id])

    if train:
        if not os.path.isfile(lmdb_store_path):
            convert(store_path + '/train', store_path + '/train.db', write_freq=1000)
    else:
        if not os.path.isfile(lmdb_store_path):
            convert(store_path + '/test', store_path + '/test.db', write_freq=1000)

    self.lmdb_store_path = lmdb_store_path
    self.lmdb_connection = lmdb.open(lmdb_store_path,
                                     subdir=False, readonly=True, lock=False, readahead=False, meminit=False)

    with self.lmdb_connection.begin(write=False) as lmdb_txn:
        self.length = lmdb_txn.stat()['entries'] - 1
        self.keys = pyarrow.deserialize(lz4framed.decompress(lmdb_txn.get(b'__keys__')))
        print("Total records: [{}, {}]".format(len(self.keys), self.length))
    self.transform = transform

def __getitem__(self, index):
    lmdb_value = None
    with self.lmdb_connection.begin(write=False) as txn:
        lmdb_value = txn.get(self.keys[index])
    #assert lmdb_value is not None, f"Read empty record for key: {self.keys[index]}"

    img_name, img_arr, img_shape = BirdDatesetLMDB.decompress_and_deserialize(lmdb_value=lmdb_value)
    image = np.frombuffer(img_arr, dtype=np.uint8).reshape(img_shape)
    if image.size == 0:
        raise InvalidFileException("Invalid file found, skipping")
    if self.bdng_box is not None:
        x, y, width, height = self.bdng_box[index]
        image = transforms.functional.resized_crop(Image.fromarray(image), y, x, height, width, (224,224))
    if self.transform:
        image = self.transform(image)

    label = torch.as_tensor(self.labels[index])
    struct = torch.as_tensor(self.struct[index]).float()

    return image, label

@staticmethod
def decompress_and_deserialize(lmdb_value: Any):
    return pyarrow.deserialize(lz4framed.decompress(lmdb_value))

def __len__(self):
    return self.length

def read_data(batch_size, valid_size=0.01, num_workers = 2):
normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
transformer = transforms.Compose([
        # transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        # transforms.ColorJitter(.4, .4, .4),
        transforms.ToTensor(),
        normalize,
    ])

test_tfms = transforms.Compose([
        transforms.Resize(int(224 * 1.14)),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        normalize,
    ])

trainset = BirdDatesetLMDB(train=True, transform=transformer)
validset = BirdDatesetLMDB(train=True, transform=test_tfms)

testset = BirdDatesetLMDB(train=False, transform=test_tfms)

num_train = len(trainset)
indices = torch.randperm(num_train).tolist()
split = int(np.floor(valid_size * num_train))

train_idx, valid_idx = indices[split:], indices[:split]
print(len(valid_idx))
train_sampler = SubsetRandomSampler(train_idx)
valid_sampler = SubsetRandomSampler(valid_idx)

train_loader = DataLoader(trainset, batch_size=batch_size, sampler=train_sampler, num_workers=num_workers, pin_memory=True)
valid_loader = DataLoader(validset, batch_size=batch_size, sampler=valid_sampler, num_workers=num_workers, pin_memory=True)

test_loader = DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=num_workers, pin_memory=True)

return train_loader, valid_loader, test_loader

def birds_collate_fn(data):
# the collate function for dataloader
collate_images = []
collate_labels = []
collate_struct = []

for d in data:
    collate_images.append(d['c'])
    collate_labels.append(d['b'])
    collate_struct.append((d['a']))
collate_images = torch.stack(collate_images, dim=0)
collate_labels = torch.stack(collate_labels, dim=0)
collate_struct = torch.stack(collate_struct, dim=0)

return {
    #'img':
        collate_images,
    #'label':
        collate_labels,
    #'struct': collate_struct
}

Context: Based on this comment 84 it seems lmdb creates this issue.


Question: how do you go back from nn.Embedding  to the original discrete values?

Context: In case you have the feature vector, you could calculate the difference and select the index with a 0 (or close to 0) diff.


Question:
Hi
How to dynamically add or delete layers during training? or how to modify the network architecture after each epoch?
Many thanks.

Context: You can do define a method on your network class that updates the architecture, and call it at the end of every epoch. The method should do something that changes how .forward behaves.


Why am I getting same output values for every single data in my ann model for multi-class classification?

Question:
In my code, i am taking a random array as a dataset. Each row of array has 4 values, and each row is one data. So if total no. of rows is suppose, 10000, then i have 10,000 data.
The task is to feed one row at a time to the model:
input layer- has 4 nodes for the 4 values in each row.
no. of hidden layers- 2 (for now)
output layer has 3 nodes for 3 classes.
Class labels are 0,1,2.
Upon training, each output contains 3 probabilities but all 3 of them have values in same range, i.e 0.3…  . I dont understand why?
Also, in validation testing, the output of all data values(each row) is same.
I have tried many variations, but now i dont understand why am i getting same output each time.
i have pasted the code here:
import torch
import torch.nn as nn
#import torch.nn.functional as f
import torch.optim as optim
#from torch.autograd import Variable
from sklearn.utils import shuffle
from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score
from sklearn.metrics import classification_report
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import LabelEncoder

import numpy as np
import time

#_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _DEFINING DATA_ _ _ __ _ _ _ _ _ __ _ __

arr1=np.random.rand(22500,4)    #random array with 22500 rows of values; (demo dataset)
lbl2=np.ones(7500)              #matrix of 0s for class 1
lbl1=np.zeros(7500)             #matrix of 1s for class 2
lbl3=np.full(7500,2)            #matrix of 2s for class 3
lbl=np.hstack((lbl1,lbl2,lbl3))
label_encode=LabelEncoder()
int_encode=label_encode.fit_transform(lbl)
onehot_encoder=OneHotEncoder(categories='auto',sparse=False)
lbl=lbl.reshape(len(lbl),1)
lbl=onehot_encoder.fit_transform(lbl)
print(lbl)

#divide the dataset arr1 into 3 parts: 20% for test set, 20% of remaining for validation set and rest is training set
percent=0.2
data1,label1=shuffle(arr1,lbl)             #shuffle the data values before partition
no=int(round(percent*len(lbl)))            #no stores 20% of data1
#print("no.of elements after first cut: ",no)

#test_input - stores 20% of data1 as test data1
#test_label - stores corresponding labels of data points in test_input
#t_input - stores remaining 80% of data points
#t_label - stores remaining corresponding 80% class labels
#valid_input - stores 20% of remaining data as validation dataset
# valid_label - stores corresponding validation set labels
#train_input - stores training dataset
#train_label - stores training labels

test_input,t_input=data1[:no],data1[no:]
test_label,t_label=label1[:no],label1[no:]
noele=int(round(percent*len(t_input)))       #stores 20% of remaining data
#print("no.of elements after second cut: ",noele)
valid_input,train_input=t_input[:noele],t_input[noele:]
valid_label,train_label=t_label[:noele],t_label[noele:]

print("size of training set: {} \n size of test set: {} \n size of validation set: {}".format(len(train_input),len(test_input),
        len(valid_input)))

#converting all datasets into torch tensors
trdata=[torch.tensor([line],dtype=torch.float) for line in train_input]
trlabel=[torch.tensor([line],dtype=torch.float) for line in train_label]
vdata=[torch.tensor([line],dtype=torch.float) for line in valid_input]
vlabel=[torch.tensor([line],dtype=torch.float) for line in valid_label]
tedata=[torch.tensor([line],dtype=torch.float) for line in test_input]
telabel=[torch.tensor([line],dtype=torch.float) for line in test_label]
#print("type of label is: ",vlabel[0].dtype)
#print("type of training label is: ",trlabel[0].dtype)
#print("type of validation dataset is: " ,type(vlabel))
#print("type of training dataset is: ", type(trdata))

#_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _DEFINING THE NETWORK_ _ _ _ _ _ _ ___ __ _ _ _ _ _ _ _ _ _

#Two layers network class
class Network(nn.Module):
    def __init__(self):
        super(Network,self).__init__()
        self.block1=nn.Sequential(nn.Linear(4,20),
                                   #nn.BatchNorm1d(20),
                                   nn.Dropout(p=0.25),
                                   nn.ReLU(),
                                   nn.Linear(20,20),
                                   #nn.BatchNorm1d(20),
                                   nn.Dropout(p=0.25),
                                   nn.ReLU())
        self.block2=nn.Sequential( nn.Linear(20,20),
                                   #nn.BatchNorm1d(20),
                                   nn.Dropout(p=0.25),
                                   nn.ReLU(),
                                   nn.Linear(20,3),
                                   nn.Softmax(dim=1))
    #forward pass
    def forward(self,x):
        x=self.block1(x)
        output=self.block2(x)
        #return f.softmax(x,dim=1)
        return output

net=Network()
#state CrossEntropyLoss as loss function; stochastic gradient descent as optimizer,
#learning rate, momentum and weight decay for regularization
criterion=nn.BCEWithLogitsLoss()
opt=optim.SGD(net.parameters(),lr=0.001,momentum=0.9,weight_decay=0.1)

#_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _TRAINING DATA_ _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _ _ _

#train() computes loss over a single data passed to it.
def train(input,target):
    output=net(input)

    print("training data is : ",input)
    print("training output is : ",output[0])
    print("\n")

    loss=criterion(output,target)
    #print("loss is : ",loss)
    opt.zero_grad()
    loss.backward()
    opt.step()
    return loss.item()

def trainer(epoch,data,target):
    net.train()
    #batch_loss stores sum of loss values computed for each data in the entire
    #data set in 1 epoch.
    batch_loss=0.0
    for iter in range(totalbatches):
        #creating batches of size batch_size from the data.
        data1=data[iter*batch_size:(iter+1)*batch_size]
        target1=target[iter*batch_size:(iter+1)*batch_size]
        #print("in trainer : ",target1[0].dtype)
        #losses stores loss of each data from a batch; returned by train()
        losses=[]
        for i in range(len(data1)):
            losses.append(train(data1[i],target1[i]))
        #print("loss for data {} :".format(i+1),loss)
        batch_loss+=sum(losses)
        #print('[%d/%d %4d] train loss: %.5f'%(epoch+1,epochs,(iter+1)*batch_size,sum(loss)/batch_size))
        print("batch %d of epoch %d successful"%(iter+1,epoch+1))
    return(batch_loss/len(data))        #return avg error over entire dataset in 1 epoch

#_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _VALIDATION TESTING_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __

def validation(epoch,vdata,vtarget):
    net.eval()
    cor=0                   #no. of correct predictions
    incor=0                 #no. of incorrect predictions
    valid_loss=0.0          #stores sum of losses over entire dataset
    op=[]                   #stores predicted class of each data
    tar=[]                  #stores target class of each data
    #print(("before getting inside loop, type is: ",vtarget[0].dtype))
    for iter in range(vbatches):
        vdata1=vdata[iter*batch_size:(iter+1)*batch_size]
        vtarget1=vtarget[iter*batch_size:(iter+1)*batch_size]
        #print(("inside loop, type is: ",vtarget1[0].dtype))
        losses=[]           #stores loss of every data in 1 batch
        for i in range(len(vtarget1)):
            output=net(vdata1[i])
            print("validset data \t:",vdata1[i])
            print("validset output \t:",output)
            print("\n")
            op.append(torch.argmax(output).item())
            tar.append(torch.argmax(vtarget1[i]).item())
            loss=criterion(output,vtarget1[i])
            losses.append(loss)

        valid_loss+=sum(losses)
        #print('[%d %5d] valid loss: %.5f'%(epoch+1,(iter+1)*batch_size,(valid_loss/len(vdata)))
    #compute total number of correct/incorrect predictions
    for i in range(len(op)):
        if op[i]==tar[i]:
            cor+=1
        else:
            incor+=1
    valid_acc=(cor/(cor+incor))*100
    #return avg loss in validation data and validation accuracy over 1 epoch
    return ((valid_loss.item())/len(vdata)),valid_acc

#_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ TESTING DATA_ _ _ _ _ _ _ _ _ _ __ _  __ _ _ _ __ _ __

def test(tdata,tlabel):
    net.eval()
    cor=0
    incor=0
    test_op=[]                      #list to store output predictions
    test_tar=[]                     #list to store target values
    error=[]                        #stores error in class prediction of each data
    #print("label is: ")
    #print(tlabel)
    for batch in range(tbatches):
        tdata1=tdata[batch*batch_size:(batch+1)*batch_size]
        tlabel1=tlabel[batch*batch_size:(batch+1)*batch_size]
        for i in range(batch_size):
            output=net(tdata1[i])

            #print("test data is: ", tdata1[i])
            #print("test output is :", output[0])
            #print("\n")

            test_op.append(torch.argmax(output[0]).item())
            test_tar.append(torch.argmax(tlabel1[i]).item())
            error.append(criterion(output,tlabel1[i]))
        print("Batch {} testing done ".format(batch+1))
    avg_error=(sum(error)/len(tdata))    #avg error computed for test data
    #compute total correct/incorrect predictions.
    for i in range(len(test_op)):
        if(test_op[i]==test_tar[i]):
            cor+=1
        else:
            incor+=1
    results=confusion_matrix(test_tar,test_op)
    print("confusion matrix:")
    print(results)
    print("acccuracy score: ")
    print(accuracy_score(test_tar,test_op,normalize=False))
    print("report :")
    print(classification_report(test_tar,test_op))
    print("average error in test set is: {}".format(avg_error))
    print("test accuracy is : {} %".format((cor/(cor+incor))*100))

# _ _ _ _ _ _ _ _ _ _ __ _ _ _ _ _ _ _DRIVER CODE_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ __ _ __ _

epochs=6
batch_size=800

trainlen=len(trdata)                    #size of training data
validlen=len(vdata)                     #size of validation data
testlen=len(tedata)                     #size of test data
totalbatches=int(trainlen/batch_size)   #total training batches
vbatches=int(validlen/batch_size)       #total validation batches
tbatches=int(testlen/batch_size)        #total test batches

t=time.time()
#train_loss stores avg loss computed over training set in every epoch
#valid_loss stores avg loss computed over validation set in every epoch
train_loss=[]
valid_loss=[]

for epoch in range(epochs):
    tloss=0.0                       #stores avg training loss returned in 1 epoch
    vloss=0.0                   #stores avg validation loss returned in 1 epoch
    #indices=torch.randperm(trainlen)
    #print(indices)
    #data,label=data[indices],label[indices]
    trdata,trlabel=shuffle(trdata,trlabel,random_state=0)
    #print(len(trdata))
    vdata,vlabel=shuffle(vdata,vlabel,random_state=0)
    t0=time.time()
    tloss=trainer(epoch,trdata,trlabel)
    vloss,valid_acc=validation(epoch,vdata,vlabel)
    #print("epoch {} successful : {} seconds ".format(epoch+1,round((time.time()-t0),3)), end=' ')
    #print("train loss: {}".format(round(tloss,5)))
    #print("validation loss: {}".format(round(vloss,5)))

    print("epoch %d/%d took %.5f seconds" %((epoch+1),epochs,round((time.time()-t0),3)))
    print("training loss: ",round(tloss,5))
    print("validation loss :",round(vloss,5))
    print("validation accuracy is: {}%".format(valid_acc))
    train_loss.append(tloss)
    valid_loss.append(vloss)

print("total training time is {} minutes : ".format(round((time.time()-t)/60,3)))
#print("train loss \t validation loss: ")
#for i in range(len(train_loss)):
#    print(round(train_loss[i],5),'\t', round(valid_loss[i],5))

#print(vlabel)

#testing data
#test(tedata,telabel)

Here is the validation result:
validset data   : tensor([[0.4604, 0.5948, 0.1612, 0.2271]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.7745, 0.0985, 0.1097, 0.4409]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.2225, 0.8885, 0.1667, 0.9272]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.4683, 0.1766, 0.2229, 0.3912]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.7469, 0.3891, 0.7210, 0.6673]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.5424, 0.6587, 0.4722, 0.3075]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.3186, 0.5349, 0.9189, 0.7124]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.7580, 0.7256, 0.3102, 0.7087]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.1943, 0.7328, 0.1706, 0.8615]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

validset data   : tensor([[0.1238, 0.6982, 0.0876, 0.9440]])
validset output         : tensor([[0.3381, 0.3278, 0.3341]], grad_fn=<SoftmaxBackward>)

Also, i changed the loss function to CrossEntropyLoss() and used class labels as 0,1,2. But the result was same.
PLz help.

Context:
Vairagi:

from sklearn.utils import shuffle from sklearn.metrics import confusion_matrix from sklearn.metrics import accuracy_score from sklearn.metrics import classification_report

I’ve gotten similar results once. In my case it was because the learning rate was so high, everything diverged and the model just gave equal probabilities for all outputs. I’d advise you to try to overfit to a small training set with a low learning rate, 10⁻5 perhaps


How to encode labels for classification on custom dataset

Question:
I am performing classification to identify which phenotype does the person in the image belong to.
In all there are eight classes
My dataset is organized as follows
Images
Character_class(contains .txt files,each txt file tells us about which class does the image belong to].The label looks like this
m_la01

No of images 800
No of labels corresponding to the images 800
As such ,given my current dataset which is shown below,what changes should I make to my dataloader so that I can train the classifier on it using standard classification loss(crossentropy).Do i need to encode the label in one hot form and return that as a tensor?
My custom dataset
class BlenderPoseDataset(Dataset):

    def __init__(self,paths,batch_size=16):
        self.img_dir='blender_data_1/images_blender/*'
        self.pose_files=paths
        self.batch_size=batch_size
        self.transforms=None
        self.image_shape=(224,224)
        self.data=[]
        self.min_data=np.load('char_min_data.npy')
        self.max_data=np.load('char_max_data.npy')
        img_list=glob.glob(self.img_dir)
        for img_path in img_list:
            for file in self.pose_files:
                self.data.append([img_path,file])
    def __len__(self):
        return len(self.data)

    def image_process(self,rgb_img):
        rgb_img=np.transpose(rgb_img.astype('float32'),(2,0,1))/255.0
        return rgb_img

    def __getitem__(self,idx):
        if self.transforms is not None:
            self.image=self.transform(self.image)
        img_path,label_path=self.data[idx]
        img=cv2.imread(img_path)
        img=cv2.resize(img,self.image_shape)[:,:,::-1].astype(np.float32)
        img=self.image_process(img)
        img=torch.from_numpy(img).float()
       with open(label_path,'r') as f:
              label=f.read(label_path)
        return img,label

dataset=Dataset(labels_paths,batch_size=32)

I have also implemented my own representation of one hot encoding that converts the class label to one hot and returns them,which can be later on used during training.Please let me know if that approach sounds reaosnable
label_arr=np.zeros(8)
classes=['f_af01','f_as01','f_ca01','f_la01','m_af01','m_as01','m_ca01','m_la01']
with open(file_name,'r') as f:
    label=f.readline()
print(label)
for i in range(len(label_arr)):
    if label==classes[i]:
        label_arr[i]=1

Context:
nn.CrossEntropyLoss can be used with a target containing class indices and starting from the latest release also accepts targets containing probabilities.
In the former case the model output should have the shape [batch_size, nb_classes] containing logits and the targets [batch_size] containing class indices in the range [0, nb_classes-1].
One-hot encoded targets could be seen as a probability, and should be accepted now, too.


Question:
Hi,
In Python I can get the PyTorch version string with:
print(torch.__version__)
However, I did not figure out a way to do the same with the C++ libtorch interface?
Does someone know how to retrieve the libtorch version string using the C++ interface?
Thanks!
Thomas

Context: Was wondering about the same. Did you find out whether this is possible yet?