foldl/torch-forum huggingface dataset:
Saving custom models

Anmol6: I have defined a custom model as follows: <SCODE>class model(torch.nn.Module):
	"""Defines custom model
	"""
	def __init__(self, dim_input, dim_output):
		
		super(model, self).__init__()
		self._dim_input = dim_input
		self._dim_output = dim_output
		
		'''Initialize nnet layers'''
		self._l1 = torch.nn.Linear(self._dim_input, SIZE_H1)
		self._l2 = torch.nn.Linear(SIZE_H2, SIZE_H1)
	self._l3 = torch.nn.Linear(SIZE_H3, SIZE_H2)


	def forward(self,x):
		self._l1_out = nn.ReLU(self._l1(x))
		self._l2_out = nn.ReLU(self._l2(self._l1_out))
		self._l3_out = nn.BatchNorm1d(self._l3(self._l2_out))
		self._out = nn.Sigmoid(self._l3_out)

		return self._out
<ECODE> How do I save this as a .t7 file? Thanks

alexis-jacq: 

Anmol6: Kinda embarrassed, but I had the order of the filename and the module reversed in torch.save()… Thanks anyway!

Anmol6: Spoke too soon - I tried saving it as a .t7 file but I can’t load it with load_lua

Anmol6: I want to load this like I would load a Sequential model. Can I not do that?

apaszke: 

Anmol6: Oh, do you have any suggestions then on how I can best load/save this custom model in python? Should I just do my best to just write my model as a nn module (eg. using Sequential) and load/save with torch.save() and load_lua respectively?

Anmol6: nvm, got it - thanks!

apaszke: 

Russel_Russel: 

apaszke: 

devansh20la: Thanks

Rao_Shivansh: Hey , I am beginner and was trying to save parameters of a pretrained network in hdf5 file and wanted to load it in torch , but unsuccessfull . Could you please let me know how to save parameters of a pretrained network (which is in pytorch ) in hdf5 file . Thanks!

Erik_Janezic: I can tell you how I saved the model and it worked import pickle This saved the model, then when I started training the model instead of starting from zero I loaded the saved model: and loaded the state of the saved model to a already initialized model: shared_model.load_state_dict(existing_model.state_dict()) The training continued from where I stopped the training.

Rao_Shivansh: does it save the model in pytorch and load it in lua ?

Erik_Janezic: 

Rao_Shivansh: Thanks !

Rao_Shivansh: Thanks a lot !

Rose: Hey, Rao, I know this post is a few months old, but were you ever able to package the model to HDF5? Thanks.
RunTime Error, Cuda

srv902: Hi, I am training a pretrained model resnet-18 imported from torchvision.models with dataset containing 1050 images of size 3x240x320. After training, I am testing with 399 samples, but I am getting RunTime CUDA Error : Out of Memory. Also, I have ported the test dataset to CUDA and volatile attribute is set to True. The model is in GPU and after training nvidia-smi output is 3243MiB/4038MiB Is there any way available to free the GPU memory so that I can do the testing?

smth: it’s possible that moving both the training and test dataset over to CUDA doesn’t give enough space for the network to forward-prop through. Loading your dataset into GPU memory itself takes 1.3 GB of memory. Are you giving a large batch size as input to your network? Can you reduce the batch size at inference time?

srv902: Yes I was giving a large batch size, so I reduced the batch size at inference time. It is working fine now. Thank you very much.

apaszke:
Dropout for RNNs

supakjk: I wonder if there would be an elegant way to use the same dropout mask on sequences for RNNs, or it would be better to implement a module. (Dropout option in the current RNN module just regard the entire sequence output as a single output. Right?) Thanks!

apaszke: You could write your own module, where you process the whole sequence, sample the mask once at the beginning and just do an element-wise multiplication after each step. Here’s a gist: <SCODE>def forward(self, x):
    mask = Variable(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout).bernoulli_())
    hidden = None
    outputs = []
    for t in x.size(1):
        output, hidden = self.rnn(x[:, t], hidden)
        outputs.append(output * mask)
    return outputs, hidden
<ECODE>

apaszke: Regarding the current RNN modules, they use a different mask for each time step.

ngimel: Performance-wise, running rnn on the whole input sequence, expanding mask and applying it to the whole rnn output will probably be better than having a loop over time.

emanjavacas: 

apaszke: 

emanjavacas: 

apaszke: Yes, if a separate mask for each timestep is ok, then you can just use the built in modules. They will be the fastest.

supakjk: <SCODE>def _make_noise:
	return input.new().resize_(1, input.size(1), input.size(2))'<ECODE>

apaszke: 

supakjk: What I did to use the same dropout mask for different time steps was inheriting classes as follows: <SCODE>class SeqConstDropoutFunc(torch.nn._functions.dropout.Dropout):
        def __init__(self, p=0.5, train=False, inplace=False):
                super(SeqConstDropoutFunc, self).__init__(p, train, inplace)

        def _make_noise(self, input):   # for timesteps X batches X dims inputs, let each time step has the same dropout mask
                return input.new().resize_(1, input.size(1), input.size(2))

class SeqConstDropout(nn.Dropout):
        def __init__(self, p=0.5, inplace=False):
                super(SeqConstDropout, self).__init__(p, inplace)

        def forward(self, input):
                return SeqConstDropoutFunc(self.p, self.training, self.inplace)(input)
<ECODE> Thanks.

apaszke: Yes I think it’s best to reimplement it yourself. It’s a very simple function.

seba-1511: I think you should sample the bernoulli using: <SCODE>Variable(torch.bernoulli(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout)))
<ECODE>

cosmozhang1988: Is there an elegant implementation of it already? I guess the idea of using the same dropout mask is from “A Theoretically Grounded Application of Dropout in Recurrent Neural Networks”.

cosmozhang1988: Did you get it implemented in pytorch already?

026f9ee49b7010c38742: Variable(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout).bernoulli_()) is not correct. The correct form is Variable(torch.Tensor(x.size(0), self.hidden_size).fill_(self.dropout).bernoulli())
Accessing intermediate data in nn.Sequential

hyqneuron: I’d like to access the output of a module inside an nn.Sequential. It could be that I have a big CNN wrapped in nn.Sequential, and I’d like to concat the output of several layers. Or it could be that I’m just checking the intermediate layer outputs to debug stuff. Doing this used to be quite easy in lua torch because modules cache their output in module.output. But pytorch apparently doesn’t do that anymore. Is there a way I can access intermediate layers’ output elegantly? Currently I just inherit from nn.Sequential and in forward() I cache the outputs from specific modules I want. I’m not sure if this screws up memory management since it looks like the intermediate values need to be re-allocated at every network pass?

smth: Subclassing sequential and overriding forward (just like you are doing now) is the right approach. We wanted to get rid of Sequential itself, but we kept it as a convenience container. There is no screwing up of memory management because of subclassing sequential (or caching outputs), it works out fine.

hyqneuron: Would this increase memory usage by prolonging the lifetime of output? I’m very curious about how intermediate values are allocated and deallocated.

smth: yes if you do that, all output variables will be cached and you will run out of memory (because in pytorch output buffers are not reused but recreated)

apaszke: You won’t necessarily run out of memory, because if you overwrite them at every forward, you’ll be keeping at most one additional copy of the graph, that’s likely to have its buffers freed if only you called backward on it.
How can i compute 3D tensor * 2D tensor multiplication?

yunjey: In PyTorch, i can do this as below. <SCODE>result = torch.mm(X.view(-1, C), Y)
result = result.view(-1, B, D)
<ECODE> Is there simpler way to handle this?

chenyuntc: <SCODE>Y.unsqueeze(0).expand_as(X) * X<ECODE>

yunjey: Y.unsqueeze(0) has a shape (1, C, D) so that it can not be expanded as X which has a shape (A, B, C). What i want to do is to broadcast 2D matrix multiplication as below. <SCODE>def matmul(X, Y):
    results = [] 
    for i in range(X.size(0)):
        result = torch.mm(X[i], Y)
        results.append(result)
    return torch.cat(results)<ECODE>

smth: 

yunjey: Ok, I got it. Thanks.

alexis-jacq: result = torch.mm(X.resize_(A*B,C), Y).resize_(A,B,D)

chenyuntc: My mistake, I was careless

jphoward: 

apaszke: Also, you be able to do that with batched matrix multiply: <SCODE>result = torch.bmm(X, Y.unsqueeze(0).expand(X.size(0), *Y.size()))
<ECODE> We’ll be adding broadcasting very soon.

apaszke: 

ngimel: 

o2h4n:
How can I modify some parts of the saved model?

Seungyoung_Park: My original model as below: <SCODE>class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.encoder = Encoder()
        self.decoder = Decoder()

    def forward(self, x):
        x = self.encoder(x)
        x = F.dropout(x, p = 0.2)
        x = self.decoder(x)
        return x
<ECODE> <SCODE>net = torch.load('net.pth')
class classifier(nn.Module):
    super(classifier, self),__init__()
    self.encoder = nn.Sequential(*list(net.encoder.children()))
    self.fc = nn.Linear(enc_dim, num_class)

    def forward(self, x):
        x = self.encoder(x)
        x = F.dropout(x,p = 0.5)
        x = self.fc(x)
        x = F.log_softmax(x)
        return x
<ECODE> Can you please tell me how I can solve this error ? Also, when I use DataParallel, I think <SCODE>nn.Sequential(*list(ae_net.encoder.children())) 
<ECODE> needs to be modified as like <SCODE>nn.Sequential(*list(ae_net.module.encoder.children()))
<ECODE> Is it right ?

smth: invalid device ordinal points to you trying to use a CUDA device id that doesn’t exist. For example, trying to use id=2 on a device with only 2 GPUs (0, 1)

Seungyoung_Park: I do not understand your explanation. Where can i add gpu id ? I have 4gpus, but at this time i just use only one by using CUDA_VISIBLE_DEVICES.

smth: CUDA_VISIBLE_DEVICES is 0-indexed, so if you have 4 GPUs, then valid values are 0, 1, 2 or 3. pytorch (actually CUDA) is complaining that you gave an invalid device id to use

apaszke: 

Seungyoung_Park: No, I tried it using CUDA_DEVICES_VISIBLE=1. But, may be i used different gpu when the model was loaded.

Seungyoung_Park: 

apaszke: 

Seungyoung_Park: As you suggested, it works! Thanks.
How to use condition flow?

t3476: 

alexis-jacq: You probably get this error when you try to use the ByteTensor as a boolean: RuntimeError: bool value of non-empty torch.ByteTensor objects is ambiguous

t3476: Does that numpy bool create proper computing graph? If so, why that’s the case? I’m not sure why pytorch can create computing graph with mixing numpy arrays and pytorch variables.

smth: Can you elaborate a bit more on what exactly you are trying to do?

apaszke: <SCODE>def forward(self, x):
    x = self.module1(x)
    if (x.data > 0).all():
        return self.module2(x)
    else:
        return self.module3(x)
<ECODE>

t3476: Problem sovled. Thanks for help.

rahul: Hi, does the condition work on a sample by sample basis? or does the condition apply across all the samples in batch? I am confused about this. Can you please explain it?

madmaxiwnl: I don’t think it works on a per sample basis. Can someone please provide a proper equivalent to tf.cond().

ptrblck: 

madmaxiwnl: Well, it’s something like this: <SCODE>out_tensor1, out_tensor2, out_tensor3 = tf.cond(some_condition,
                         lambda: some_tensor1, some_tensor2, some_tensor3,
                         lambda: some_tensor4, some_tensor5, some_tensor6)
<ECODE> Currently, I am using a workaround I found on another thread: <SCODE>def where(cond, f1, f2):
    return c * f1() + (1-c) * f2()
<ECODE> and finding all 3 out_tensors seperately. Though having an inbuilt operation in pytorch would be useful.

ptrblck: I’m not sure, how the TF code works exactly, but if it’s just an assignment, wouldn’t this work: <SCODE>if some_condition:
    out_tensor1, out_tensor2, out_tensor3 = some_tensor1, some_tensor2, some_tensor3
else:
    out_tensor1, out_tensor2, out_tensor3 = some_tensor4, some_tensor5, some_tensor6
<ECODE>

madmaxiwnl: <SCODE>some_condition = some_condition.type(torch.LongTensor)
out_tensor1 = some_condition * some_tensor1 + (1-some_condition) * some_tensor2
<ECODE> and, similarly for out_tensor 2 and 3. I couldn’t find a function which would perform the operation for all 3 out_tensors at once. Anyways it’s working now, thanks.

abediee: Use

wiiiktor: Could the tensor.unbind() function be used on .data, to work per sample basis?
Is there a way to mix many different activation functions efficiently?

Veril: Suppose I want a layer to consist of 10 different activation functions. Right now the only way to do this seems to basically involve the creation of multiple layers and concatenating them. This obviously introduces some computational inefficiency. Is there some clever hack to apply point-wise nonlinearities based on masks or something? The way I envision this possibly happening is by doing the linear tranformation as usual  but then stacking multiple nonlinearities on top such that each nonlinearity layer ignores all but its own portion of the layer. Not sure how to approach that.

csarofeen: Activation functions or general pointwise functions? Everything applied at the same time on the same tensor or a string of pointwise functions on subtensors?

Veril: Not sure about what distinction you are making between activation functions and “general pointwise functions”. The goal is basically have different neurons have different non-linearities. Not apply different functions to the same neurons.

csarofeen: So why not have a series of tensors with value 0 or 1 to pre-multiply the input before each activation and accumulate in an output?

Veril: Would that be an efficient solution? If I understood you correctly, you would have a lot of function operations with input as zero, or would that be optimized away?

apaszke: That won’t be optimized away. If you really want to optimize it this should work: <SCODE>nonlinearities = [F.relu, F.tanh, F.sigmoid]
masks = generate_masks(input) # a list of masks
preactivation = module(input) 
# preallocate output
# IMPORTANT: it shouldn't require grad!
# you don't care about the grad w.r.t the original content!
output = Variable(preactivation.new(preactivation.size()))
for nonlinearity, mask in zip(nonlinearities, masks):
    output[mask] = nonlinearity(preactivation[mask])
<ECODE>

apaszke: 

wxwx: hello, i have the same question, Have you solved this problem?

awalkingstick: It sounds like I’ll need to implement a new mixed layer to replace an existing ReLU layer. In my new mixed layer I’ll need to generate a set of masks for each activation function I intend to use. Then I need to pass each masked input to the corresponding activation function, and assign the output of these to the corresponding masked output. Does that all sound correct/possible?

awalkingstick:
How can i load coco caption data set using prebuilt tools?

yunjey: <SCODE>import torchvision.datasets as dset
import torchvision.transforms as transforms

cap = dset.CocoCaptions(root = '../data/train2014',
                        annFile = '../data/annotations/captions_train2014.json',
                        transform=transforms.ToTensor())

print('Number of samples: ', len(cap))
img, target = cap[3] # load 4th sample

print("Image Size: ", img.size())
print(target)
<ECODE> <SCODE>/Users/yunjey/JupyterProjects/kwtl-pytorch/coco/PythonAPI/pycocotools/coco.py in <module>()
     53 import copy
     54 import itertools
---> 55 from . import mask as maskUtils
     56 import os
     57 from collections import defaultdict

/Users/yunjey/JupyterProjects/kwtl-pytorch/coco/PythonAPI/pycocotools/mask.py in <module>()
      1 __author__ = 'tsungyi'
      2 
----> 3 import pycocotools._mask as _mask
      4 
      5 # Interface for manipulating masks stored in RLE format.

ImportError: No module named _mask
<ECODE> Is there a simple way to install the coco api to solve this problem? Or, which directory should i clone the coco api repo?

yunjey: I have solved this problem by running make inside the coco/PythonAPI.  Still remaining problem is that i should run the code inside coco/PythonAPI, this is inconvenient. (from pycocotools.coco import COCO)

Response777: 

Response777: 

yunjey: Thanks for your reply. I am using Python 2.7.

Response777: 
The first problem you came across is the same with https://github.com/pdollar/coco/issues/14 180, which is caused by forgetting make.(The cython was not compiled.)
 
The second problem is that this package is not in the path your python script would search in.

You may solve this running the setup.py in PythonAPI folder to copy this package to your pip folder.
Or you can add such code in your script```
import sys
sys.path.append(“The path of PythonAPI folder”)

 You may solve this running the setup.py in PythonAPI folder to copy this package to your pip folder. Or you can add such code in your script```
import sys
sys.path.append(“The path of PythonAPI folder”) <SCODE><ECODE>

yunjey: Thank you very much! Your reply helped a lot.

apaszke: 

Vyvian_Nellira: It still fails for me. these are the following steps I took
torch.utils.DataLoader causes a RuntimeError

yunjey: <SCODE>import torch
import torchvision.datasets as dset
import torchvision.transforms as transforms

cap = dset.CocoCaptions(root = './data/train2014resized',
                        annFile = './data/annotations/captions_train2014.json',
                        transform=transforms.ToTensor())

print('Number of samples: ', len(cap))
img, target = cap[3]    # this works well

train_loader = torch.utils.data.DataLoader(
    cap, batch_size=1, shuffle=False, num_workers=1)
data_iter = iter(train_loader)

print (data_iter.next())    # this returns an error.
<ECODE> When I ran the code above, i got a huge RuntimeError message. The below is the bottom of the error message. <SCODE>File "/Users/yunjey/anaconda2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 75, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/Users/yunjey/anaconda2/lib/python2.7/site-packages/torch/utils/data/dataloader.py", line 71, in default_collate
    elif isinstance(batch[0], collections.Iterable):
  File "/Users/yunjey/anaconda2/lib/python2.7/abc.py", line 132, in __instancecheck__
    if subclass is not None and subclass in cls._abc_cache:
  File "/Users/yunjey/anaconda2/lib/python2.7/_weakrefset.py", line 75, in __contains__
    return wr in self.data
RuntimeError: maximum recursion depth exceeded in cmp
<ECODE> What can i do for solving this problem?

apaszke: Do you have unicode objects in your batches?

yunjey: I solved this problem by converting return type unicode to string. Could i pull request to solve this problem?

smth: If you could send a pull request for (1), that would be great. Thank you.

apaszke: 

yunjey:
Where should i implement the preprocessing code for text? (e.g seq2seq learning)

yunjey: I want to preprocess the text data, for example, converting each word to index and adding some pads (for seq2seq learning). Is it good way to handle this as below? <SCODE>   class MyDataset(torch.data.utils.Dataset):
        def __init__(self):
            self.data_files = os.listdir('data_dir')
            sort(self.data_files)

        def __getitem__(self, idx):
            data =  load_file(self.data_files[idx])
            data = preprocess_data(data)            # preprocess
            return data 

        def __len__(self):
            return len(self.data_files)


dset = MyDataset()
loader = torch.data.utils.DataLoader(dset, num_workers=8)<ECODE>

smth: yes, this is a good way to handle text loading.

Spider101: The repo looks really good. Is there a potential timeline on when it might be released on pip?
Extension of convolution layer using CFFI

Ja-Keoung_Koo: It seems that I can start from some sources from pytorch/torch/lib/… Could I just know what sources I have to use or a big picture of getting started?

smth: What additional information would get you started?

apaszke: 

Ja-Keoung_Koo: Thanks for your information. In /THNN/docs I found a description of module = nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH], [padW], [padH]) But it seems that there is no SpatialConvolution in THNN/generic. I am just curious:

smth: 

Zijun_Wei: <SCODE>THNN_(col2im)(
  THTensor_(data)(gradColumns),
  nInputPlane, inputHeight, inputWidth, outputHeight, outputWidth,
  kH, kW, padH, padW, dH, dW,
  dilationH, dilationW,
  THTensor_(data)(gradInput_n)
);
<ECODE>

tom: Hi, Best regards Thomas

Zijun_Wei: 

tom: 

Zijun_Wei: THNN_Floatim2col(THFloatTensor_data(input_h_w), 1, inputHeight, inputWidth, kH, kW, padH, padW, dH, dW, dilationH, dilationW, THFloatTensor_data(columns)); But when I compile the file, the warning comes out: <SCODE>warning: implicit declaration of function 'THNN_Float_im2col' is invalid in C99 [-Wimplicit-function-declaration]
            THNN_Float_im2col(THFloatTensor_data(input_h_w), 1, inputHeight, inputWidth, kH, kW, padH, padW, dH, dW, dilationH, dilationW, THFloatTensor_data(columns))
<ECODE> I wonder this might because I need to declare it first. So the question is,  what is the full declaration for THNN_Float_im2col? I’m new to c, so if I understand this wrong or there is an other way to solve this problem, please let me know. Many thanks.

tom: I think the declaration is more <SCODE>static void THNN_Floatim2col(const float* data_im, const int channels,
      const int height, const int width, const int kernel_h, const int kernel_w,
      const int pad_h, const int pad_w,
      const int stride_h, const int stride_w,
      const int dilation_h, const int dilation_w,
      float* data_col);
<ECODE> or so (by literally copying the declaration in the file and substituting). From the message, you seem to have an extra underscore somewhere. Best regards Thomas

Zijun_Wei: Where did you copy this?

tom: Hi! Best regards Thomas

Zijun_Wei: THNN_Floatim2col' has internal linkage but is not defined If I removed the declaration I got: implicit declaration of function 'THNN_Floatim2col' is invalid in C99 I will continue to check if it is possible to get the header files that defines THNN_Floatim2col.

Zijun_Wei: 

111128: 

mhubii: