foldl/torch-forum huggingface dataset:
How to parallel in GPU when finetuning

chenyuntc: In my Finetune models , I wanna parallel my model, in multi-gpus,  my code is shown below: <SCODE>class FinetuneModel(nn.Module):
    def __init__(self, pretrained_model, ngpu = opt.gpuids):
        self.ngpu = ngpu
        super(FinetuneModel, self).__init__()
        self.features = pretrained_model
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512 * 4 * 4, 2048),
            ....           
))      
        
    def forward(self, x):
        gpuids = None
        if self.ngpu:
            gpuids = range(self.ngpu)
        features =  self.features(x)#self.features has already implemented data parallel
        return nn.parallel.data_parallel(self.classifier, features, device_ids=gpuids)

<ECODE> as far as I know , when doing <SCODE>features =  self.features(x)#self.features.forward has already implemented data parallel
score = nn.parallel.data_parallel(self.classifier, features, device_ids = gpuids)
<ECODE> <SCODE>score = nn.parallel.data_parallel([self.features,self.classifier], features, device_ids = gpuids)
<ECODE> which only does one broadcast

Cysu: 

chenyuntc: 

Cysu: As AlexNet and VGG contain lots of parameters in the FC layers. Syncing params on these layers will have large overhead. It’s faster to compute the FC layers only on one GPU.

chenyuntc: I see. great thanks.

apaszke:
Quickly loading raw binary files

abweiss: I need to load raw int16 binary files on-the-fly during training.  In Lua Torch I could just do the following: <SCODE>img = torch.ShortStorage(filename)
<ECODE> It appears that that functionality does not exist in PyTorch.  So instead I am loading into a numpy array as follows: <SCODE>img = np.fromfile(filename, 'int16')
<ECODE> The problem is that np.fromfile is extremely slow.  For my large data files (>100MB), np.fromfile loads four orders of magnitude slower than the old torch.ShortStorage method.  How can I get the fast load speeds in PyTorch?

apaszke: 

abweiss: Apparently numpy has (fast) memory mapping: <SCODE>img = np.memmap(filename, dtype='int16', mode='r').__array__()
<ECODE>

apaszke: 

abweiss: RuntimeError:  can’t convert a given np.ndarray to a tensor - it has an invalid type.  The only supported types are: double, float, int64, int32, and uint8.

apaszke: Huh, we should add int16 to. I’ll open an issue.
View Output in PyTorch

srv902: Hi, Is there a way to view output images of the model in PyTorch other than converting to numpy and viewing with matplotlib? Help please.

apaszke: How are you displaying the images from Keras? Maybe they’re in an incorrect range

srv902: Hi, For Keras I am using simple Matplotlib functions. The range is same in both frameworks. The problem was with the dataset and the salt and pepper noise diminishes if I increase the number of epochs. It worked in Keras, but haven’t checked in PyTorch yet.
Convergence speed difference compared to that of Torch7

Seungyoung_Park: I implemented the same autoencoders with BCE criterion using pytorch and torch7. When I tried to train them using SGD with momentum, the convergence speeds were almost same within about 2000 iterations. But, the convergence of pytorch becomes quite slower than that of torch7. Has anyone compared the convergence performances ?

smth: 

Seungyoung_Park: I also tried rmsprop with batch normalization, the same things were observed. The below is a modified rmsprop employed in Torch7. <SCODE>square_avg:mul(alpha)
square_avg:addcmul(1.0-alpha, grad, grad)
avg:sqrt(square_avg+eps)
params:mul(1-lr*weight_decay):addcdiv(-lr, grad, avg)
<ECODE> And, the below is an equivalent rmsprop in pytorch <SCODE>square_avg.mul_(alpha).addcmul_(1 - alpha, grad, grad)
avg = square_avg.add(group['eps']).sqrt()
p.data.mul_(1-group['lr']*weight_decay).addcdiv_(-group['lr'], grad, avg)
<ECODE> Are they equivalent ? Also, my pytorch code didn’t work at all before 0.19 version due to some run time errors. Is that related to this convergence difference ?

apaszke: 

Seungyoung_Park: Thanks for your reply. Is it possible that the convergence becomes slower even if some gradients becomes NaNs or infs ? My model is just autoencoder consisting of convolution, maxpooling, maxunpooling, relu, and sigmoid. Anyway, to compare the gradients, I found that the gradients can be saved in a flatted form by <SCODE>torch.cat([g.grad.view(-1) for g in model.parameters()],0)<ECODE>

apaszke:
Loading VOC 2012 dataset with Dataloaders
Category: vision

Gaurav_Pandey: VOC 2012 dataset consists of images and their corresponding segmentation maps. I want to apply similar transforms to both the image and its segmentation map while loading. Any suggestions about how to proceed for this task?

apaszke: 

ncullen93: and here are some relevant affine transforms to actually use – you’ll see the transforms must take in two arguments for the input and target images:

Gaurav_Pandey: Thanks guys. That was very helpful.

bodokaiser: 

isalirezag: 

HT_Wang:
Speed deteriorates when two replicates running with same random seed

yikang-li: Hi, I met an interesting problem. When I ran two replicates of my model on two GPUs with CUDA_VISIBLE_DEVICES macro and initialized them with same random seed, the speed of the two training process deteriorates after several iterations. I guess it may be related the implementation of the ROI pooling layer. When initialized with the same seed, some GPU operations will lead conflicts. Is there anyone knowing why?

apaszke: So if you run a single process it’s ok, but if you start two it will run slower? Maybe the GPUs are competing for the PCIe bandwidth

yikang-li: Thank you very much. Maybe it is due to the competition of PCle bandwidth. But at the beginning tens of thousands of iterations, there is no such problem. Do you know what I can do to avoid the problem?

yikang-li: The problem always comes with the low usage of GPU usage.

Abhishek_Pal:
Debugging autograd functions in pytorch
Category: vision

pratheeksha: Hi! I wrote below function that I intend to use as the loss function. <SCODE>class MyCriterion(torch.autograd.Function):
    def __init__(self):
        self.alpha = .0005
    
    def forward(self, input, target, epoch, isLabeled):
        loss = F.cross_entropy(input, target)

        self.save_for_backward(input, target, epoch, isLabeled, loss)
        print(self.saved_tensors) #returns ()
        if (isLabeled.data > 0).all():

            return Variable(loss.data * self.alpha * epoch.data)
        return loss

    def backward(self, grad_output):
       
        input, target, epoch, isLabeled, loss, = self.saved_tensors
        grad_input = loss.backward() 
        return grad_input
my_criterion = MyCriterion()
x = Variable(torch.randn(11, 10).type(torch.FloatTensor))
y = Variable(torch.range(1,6, .5).type(torch.LongTensor))

a = torch.from_numpy(np.array([0]))
b = torch.from_numpy(np.array([1]))
c = torch.from_numpy(np.array([10.0]))

print(x)
# print(torch.from_numpy(np.array([10])))
first_loss  = my_criterion.forward(x, y, Variable(c.float()),  Variable(a))
print(my_criterion.backward(first_loss))

second_loss = my_criterion.forward(x, y, Variable(c.float()),  Variable(b))
print(my_criterion.backward(second_loss))
<ECODE> When I do this, I have below error - <SCODE>---> 18         input, target, epoch, isLabeled, loss, = self.saved_tensors
     19         grad_input = loss.backward()
     20         return grad_input

ValueError: not enough values to unpack (expected 5, got 0)
<ECODE> Is there something I am missing? How can I access saved tensors? Is there any other documentation than autograd that has more examples of autograd functions? Thanks a lot!

apaszke: First thing is that you can’t return a Variable from forward - it expects tensors and will automatically wrap them in Variables and connect them up to the existing graph. You shouldn’t unpack and re-pack Variables in the middle of computation, because that will break continuity of history. You need to do something like that: <SCODE>def my_criterion(input, target, epoch, isLabeled):
    if (isLabeled.data > 0).all():
        return loss * alpha * epoch
    return MyCriterion()(input, target, epoch, isLabeled)
<ECODE>

jekbradbury: 

pratheeksha: This is very useful, I will try writing it as a module, thanks!

pratheeksha: Thanks!

smth:
Introducing the TorchSample package for comprehensive data transforms and sampling
Category: vision

ncullen93: Hi all, It also has great support for situations where both the input and target tensors are images (e.g. segmentation datasets). It also supports arbitrary data types. It supersedes the currently available sampling code in the main torchvision/torch codebase. It’s my hope that this will kick-start the community-driven development of the sampling code in the main torch and torchvision packages, and serve as reliable and flexible sampling code in the meantime. NOTE: This package is in no way endorsed by, affiliated with, or otherwise associated with the official Pytorch ecosystem or team.

smth: 

ncullen93: 

Shi_Shu: Could you please give me some details on how to install this package? I have conda installed and have created an virtualenv for it. Thanks!

Rohit_Keshari: 

flock: If you are using pip, just clone, go to the directory with setup.py and run “pip install -e .” - this will read the setup and install the package (this way you can also easily uninstall it if needed)
Potential memory leak

Lifeng_Jin: We have been recently tracking some memory leak on our machine. I created a very simple script which just does this: <SCODE>a = []
for i in range(100):
    t = torch.ones(10,1000,1000)
        if cuda:
            t = t.cuda()
    a.append(t)
<ECODE> and I use a flag to control whether .cuda() is used or not. It seems that the CUDA part is causing some kind of memory leak. In the following data, the used started at 4.8G but they were dead memory that did not show up as used by any process in TOP. After running the cuda version of the loop a few times, 300M more memory were dead. I wonder if anyone has any idea as to what is happening? I am using RedHat 4.4.2, Python 2.7.13 with the newest PyTorch, CUDA 7.5.17. <SCODE>test$ free -h
             total       used       free     shared    buffers     cached
Mem:           23G       6.0G        17G       9.1M       360M       919M
-/+ buffers/cache:       4.8G        18G
Swap:          82G         0B        82G
test$ python test.py cpu
test$ python test.py cpu
test$ python test.py cpu
test$ python test.py cpu
test$ python test.py cpu
test$ free -h
             total       used       free     shared    buffers     cached
Mem:           23G       6.0G        17G       9.1M       360M       919M
-/+ buffers/cache:       4.8G        18G
Swap:          82G         0B        82G
test$ python test.py cuda
test$ python test.py cuda
test$ python test.py cuda
test$ python test.py cuda
test$ python test.py cuda
test$ free -h
             total       used       free     shared    buffers     cached
Mem:           23G       6.4G        17G       9.1M       360M       919M
-/+ buffers/cache:       5.2G        18G
Swap:          82G         0B        82G

<ECODE>

smth: Please understand that the kernel does not always have to free memory unless it thinks it needs to, it might be caching some pages for various reasons. This is not a memory leak, unless you literally cannot reclaim memory (i.e. if you try to allocate 18G + 0.4GB, and allocations fail).

Lifeng_Jin: I do understand that. This used memory is never claimed by the kernel anytime, even if it goes into swap mode because it runs out of memory when a memory intensive program is run.

smth: 

Lifeng_Jin: OK. Thank you for your help. I will upgrade CUDA and test this again.

trdavidson: 

Lifeng_Jin: It did. We did a very trivial test creating a lot of tensors either on the GPU or on the CPU and found that it was definitely the GPU. The update solved it.
Normalization of CIFAR and MNIST datasets
Category: vision

dlmacedo: <SCODE># The output of torchvision datasets are PIL Image images of range [0, 1].
# We transform them to Tensors of normalized range [-1, 1]
transform=transforms.Compose([transforms.ToTensor(),
                              transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                             ])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, 
                                          shuffle=True, num_workers=2)
<ECODE> But the MNIST example (pytorch/examples/mnist) uses values very different from 0.5 to normalize the data: <SCODE>testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4, 
                                          shuffle=False, num_workers=2)

datasets.MNIST('../data', train=True, download=True,
               transform=transforms.Compose([
                   transforms.ToTensor(),
                   transforms.Normalize((0.1307,), (0.3081,))
               ])),
batch_size=args.batch_size, shuffle=True, **kwargs)
<ECODE> Why are we using two different approaches to normalize MINIST and CIFAR? What is the correct approach to use? I guess we are NOT normalizing the dataset in the first case, but only each image. If this is the case, I think we should clarify this in the PyTorch Tutorial. Thanks, David

smth: MNIST is not natural images, it’s data distribution is quite different.

BarryBA:
Elegant implementation of Spatial Pyramid Pooling layer?

BarclayII: 

apaszke: <SCODE>def spatial_pyramid_pooling(input, output_size):
    assert input.dim() == 4 and input.size(2) == input.size(3)
    F.max_pool2d(input, kernel_size=input.size(2) // output_size)
<ECODE>

Rao_Shivansh: Hi like in torch for using spatialpyramidpooling i can directly call (inn.SpatialPyramidPooling({8,8},{4,4},{2,2},{1,1})) can i do it similarily in some way in pytorch ? What is the easiest way to do so. Thanks

addisonklinke: 

Khawar_Islam: I am implementing spatial pooling in my network but I still confuse about how to implement this? I thin there is some implementation problem in my class because my class only contain one Conv layer <SCODE>class DWConv(nn.Module):

    def spatial_pyramid_pool(self, previous_conv, previous_conv_size, out_pool_size=[4, 2, 1]):
        '''
          previous_conv: a tensor vector of previous convolution layer
          previous_conv_size: an int vector [height, width] of the matrix features size of previous convolution layer
          out_pool_size: a int vector of expected output size of max pooling layer

          returns: a tensor vector with shape [1 x n] is the concentration of multi-level pooling
        '''
        num_sample = previous_conv.shape[0]
        for i in range(len(out_pool_size)):
            h_wid = int(math.ceil(previous_conv_size[0] / out_pool_size[i]))
            w_wid = int(math.ceil(previous_conv_size[1] / out_pool_size[i]))
            h_pad = (h_wid * out_pool_size[i] - previous_conv_size[0] + 1) / 2
            w_pad = (w_wid * out_pool_size[i] - previous_conv_size[1] + 1) / 2
            maxpool = torch.nn.MaxPool2d((h_wid, w_wid), stride=(h_wid, w_wid), padding=(int(h_pad), int(w_pad)))
            x = maxpool(previous_conv)
            if (i == 0):
                spp = x.view(num_sample, -1)
            else:
                spp = torch.cat((spp, x.view(num_sample, -1)), 1)
        return spp
    
    def __init__(self, dim=768):
        super(DWConv, self).__init__()
        self.dwconv = nn.Conv2d(dim, dim, 3, 1, 1, bias=True, groups=dim)

    def forward(self, x, H, W):
        B, N, C = x.shape
        x = x.transpose(1, 2).view(B, C, H, W)
        x = self.dwconv(x)
        x = x.flatten(2).transpose(1, 2)
        x = self.spatial_pyramid_pool(previous_conv_size=x.shape)
        return x
<ECODE> Traceback <SCODE> x = self.spatial_pyramid_pool(previous_conv_size=x.shape)
TypeError: spatial_pyramid_pool() missing 1 required positional argument: 'previous_conv'
<ECODE>
Are Tensors column major or row major?

mattmacy: I’m porting some code from Caffe which column major and it would be helpful to know if torch is similarly column major or is row major.

smth: Torch Tensors are row-major.

fmassa: 

mattmacy: This “deformation_layer” isn’t part of upstream Caffe, so maybe it’s only their code to make the use of BLAS cleaner. top[0]->Reshape( out_shape);
How to create patches / windows of image dataset?

bodokaiser: I would now like to know where would be the best place to implement this type of preprocessing?

apaszke: Yeah that’s a tricky one. There’s no way to tell the DataLoader to dynamically skip some inputs and replace them with next ones, because it would violate the guarantee that it will always return the data in the same order as the given by the sampler.

bodokaiser: As you said I think I will need to add another stage which converts the data from HDF5 to patches and then feed that data into pytorch. Thanks in advance!

trypag: 

apaszke: You don’t need to save the patches ahead of time, it should be enough to store the file names and locations to retrieve them later. But if you don’t have a lot of data it might be simpler to preprocess the dataset.

bodokaiser: 

bodokaiser: 

trypag: sure, I emailed you!!

apaszke: 

bodokaiser: 

Tepp: 

trypag: See you

ncullen93: Anyways, a very simple approach to sampling 2D patches from 3D MRI is to extract a random slice (see my package torchsample for that transform) then just take a random crop. You can use the “MultiSampler” sampler in torchsample to go through each image more than once in a single epoch. No need to extract EVERY possible patch haha just do it randomly at train time. It’s still a fun exercise to reimplement them tho… which paper specifically are you interested in? I think I have implementations of most of them and can share or help you develop the right sampler.

trypag: 

ncullen93: Yeah you kinda have more data but it’s misleading bc you don’t have more region-specific subject variability. Data is definitely an issue… I train on a lot of subjects (a few thousand) from multiple large datasets… Also, the KEY to MRI segmentation is data augmentation and applying random affine transforms specifically since images differ that way across scanners - hence why I developed torchsample. With good augmentation we have done quite well with ~200 subjects.

rogetrullo: 

Meghal_Dani: 

akbarnejad: 

Najeh_Nafti: 

fepegar:
Saving torch models

Russel_Russel: <SCODE>class VAE(nn.Module):
    def __init__(self):
        super(VAE, self).__init__()

        self.fc1 = nn.Linear(784, 400)

model = VAE()

with open('model_save', 'wb') as f: 
torch.save(model, f)
<ECODE>

apaszke: I have no idea when are you getting the first error and what does it mean. I’ve never seen that.

Russel_Russel: 

Veril: <SCODE>net = Net(...).cuda()
torch.save(net.state_dict(), './net.pth')
net.load_state_dict(torch.load('./net.pth'))<ECODE>

apaszke: 

x0s: 

jphreid: I am implementing these exact same lines of code, but I get an error. Code : (On a different computer, with GPU) Code: (on my laptop) Error : module ‘torch._C’ has no attribute ‘_cuda_getDevice’ Note : I trained my model with cuda = True on a virtual environment on a network (with GPUs), but I want to evaluate my model on my laptop. I believe we need to ‘switch off’ the cuda before saving. Could you help?

jphreid: Ok, got it! …  map_location=lambda storage, loc: storage) : load the model on CPU.
Different fine-tuning results between Torch and PyTorch

panovr: I’m fine-tuning a same custom dataset using Torch and PyTorch based on ResNet 34. 
With Torch, I use fb.resnet.torch 11 with learning rate 0.001. After 50 epoch:
 
top1: 89.267  top5:  97.933
 
With PyTorch, I use code below for fine-tuing. After 90 epoch:

learning rate = 0.01, top1: 78.500, top5: 94.083
learning rate = 0.001, top1: 74.792, top5: 92.583

 learning rate = 0.01, top1: 78.500, top5: 94.083 learning rate = 0.001, top1: 74.792, top5: 92.583 You can see that PyTorch fine-tuning result is still not so good as Torch. Fine-tuning ResNet 18 has similar results. Any suggestion or guidance? PyTorch code used for fine-tuning: <SCODE>class FineTuneModel(nn.Module):
    def __init__(self, original_model, arch, num_classes):
        super(FineTuneModel, self).__init__()
            # Everything except the last linear layer
            self.features = nn.Sequential(*list(original_model.children())[:-1])
            self.classifier = nn.Sequential(
                nn.Linear(512, num_classes)
            )

        # Freeze those weights
        for p in self.features.parameters():
            p.requires_grad = False


    def forward(self, x):
        f = self.features(x)
        f = f.view(f.size(0), -1)
        y = self.classifier(f)
        return y<ECODE>

colesbury: Your fine tune model looks OK to me. I don’t have enough context to see what could be different. Can you post your fine-tuning code?

panovr: 

shicai: one possible reason is that, torch adopts more image transforms for data augmentation than pytorch.

fmassa: I think that the difference comes from the fact that you fix all the weights in the pytorch model (except from the last classifier), while in lua torch you are fine-tuning the whole network.

apaszke: 

panovr: different data augmentations or image transforms “hard” fine-tuning and “soft” fine-tuning From my dataset experiment result, I think maybe “soft” fine-tuning is better than “hard” fine-tuning.
Accessing a variable inside the model

Veril: What’s a good way to access a convolutional filter (the learned weights) in the model? edit: The filter in question nn.Conv2d(1, 1, 5, stride=1, padding=2, bias=False)

apaszke: 

Veril: That works. <SCODE>Parameter containing:
(0 ,0 ,.,.) = 
 -0.1781 -0.3750  0.0752 -0.1060  0.1356
  0.1607 -0.2711  0.1783  0.2942  0.0471
  0.1992  0.0228 -0.1627 -0.4729 -0.0560
  0.1801 -0.0715  0.0305 -0.0124 -0.1072
  0.2290  0.3730  0.1166 -0.1296  0.0746
[torch.cuda.FloatTensor of size 1x1x5x5 (GPU 0)]<ECODE>

Veril: How can you call the conv operator with non learnable tensors? Suppose I have that tensor I posted earlier that I really like, I want it to be constant and not have to worry about it being unfrozen during a p.requires_grad = True operation such as is often performed in GANs. Or I have it as a variable declared outside the class of the model where I have direct control over it from the main execution flow.

jekbradbury: 

Veril: It’s not obvious how can you force conv2d to use it?

apaszke: Wrap it in a Variable at every forward and give it to Conv2d

Veril: So I need to manually set self.conv.weight = Variable(filter) inside forward? The way you said “give it to” implies that it can be passed as an argument? I don’t see how.

jekbradbury: 

apaszke: Don’t save the variable anywhere! It should be created at every forward and use as a local Variable. I don’t really know what you want to do but I guess this might help: <SCODE>filter = Variable(self.filter) # self.filter is a registered buffer (tensor)
result = F.conv2d(input, filter)
<ECODE>