foldl/torch-forum huggingface dataset:
Properly make autograd.Function with scalar variable

Ja-Keoung_Koo: <SCODE>class mul_scalar(torch.autograd.Function):
  """
  Customized autograd.Function of
  f(T,s) = s * T,
  where T is a fixed Tensor and s is a Variable
  """

  def forward(self, T, s_var):
    self.save_for_backward(T, s_var)
    return T.mul(s_var[0])

  def backward(self, grad_output):
    T, s_var = self.saved_tensors
    return grad_output.mul(s_var[0]), grad_output.dot(T)
<ECODE> <SCODE>class Net(nn.Module):
  def __init__(self, var=1):
    super(Net, self).__init__()
    self.ms = mul_scalar()
    
  def forward(self, x):
    ...
    self.ms(x, w)
    ...
<ECODE> <SCODE>  def forward(self, x):
    c = Variable(torch.FloatTensor([1]), requires_grad = True)
    ms = mul_scalar()
    z1 = ms(x, c)
<ECODE>

fmassa: <SCODE>def mul_scalar(T, s_var):
    # supposes that T is 1D var
    # need to unsqueeze s_var if not
    return T * s_var.expand_as(T)
<ECODE>

apaszke: 

Ja-Keoung_Koo: Thanks for kind replies. Is it safe to use functions like expand_as, unsqueeze and so on? I wonder that when backprogating, grad_output is automatically resized.

fmassa: Yes, it is safe to use these functions, and they were unit tested so they should be fine.
How to Reverse a Torch Tensor

peak: How to Reverse a Torch Tensor

fmassa: <SCODE>tensor = torch.rand(10) # your tensor
# create inverted indices
idx = [i for i in range(tensor.size(0)-1, -1, -1)]
idx = torch.LongTensor(idx)
inverted_tensor = tensor.index_select(0, idx)
<ECODE>

qianguih: Hi, it looks like this can’t work on torch.autograd.variable.Variable. Is there any way to reverse a specific dimension in a Variable? Thanks!

apaszke: 

DiffEverything: If your use case is to reverse sequences to use in Bidirectional RNNs, I just create a clone and flip using numpy. <SCODE>rNpArr = np.flip(fTensor.numpy(),0).copy()   #Reverse of copy of numpy array of given tensor
rTensor = torch.from_numpy(rNpArr)          
<ECODE>

Sunil_Sharma: Can you try something like:

aRI0U: 

chorus12:
Pytorch version of SpatialFullConvolution

Spider101: I am trying to recreate a torch7 model architecture in pytorch. I have used several layers of SpatialFullConvolution and as such, was wondering if there is anything analogous to that in PyTorch. I have not been able to find anything similar by name.

smth: ConvTranspose2d

Spider101: Okay. I made the changes but now I am getting a runtime error: <SCODE>RuntimeError: input and target have different number of elements: 
input[128 x 1 x 128 x 128] has 2097152 elements, while target[128 x 2 x 128 x 128] has 4194304 element
<ECODE> This is the code for my model architecture: <SCODE>class ColorizerNet(nn.Module):

def __init__(self):
	super(ColorizerNet, self).__init__()
	self.layer1 = nn.Conv2d(1, 8, 2, 2)
	self.layer2 = nn.Conv2d(8, 16, 2, 2)
	self.layer3 = nn.ConvTranspose2d(16, 8, 2, 2)
	self.layer4 = nn.ConvTranspose2d(8, 1, 2, 2)


def forward(self, x):
	x = F.relu(self.layer1(x))
	x = F.relu(self.layer2(x))
	x = F.relu(self.layer3(x))
	x = F.relu(self.layer4(x))
	return x
<ECODE> Am I making any obvious errors here? If required, I can start a separate thread on this followup error.

apaszke: It’s an error in the loss function. Your output’s second dimension has a different size (1) than the target’s (2).
Flatten the parameters in DataParallel

Cysu: I tried the ImageNet example with ResNet152 on 8GPUs but it is much slower than fb.resnet.torch (1.5s vs 0.8s per iter). It’s elegant to implement the Broadcast as an Op/Function. I wonder if it is possible to overlap the communication with computation during forward/backward? Or it is necessary to flatten the parameters in order to improve the efficiency?

apaszke: We don’t support flattening the parameters. It’s quite complex and bug prone to do that correctly, and maintain flexibility. The main problem is that with 8 GPUs we’re a bit limited because of Python’s GIL (only one thread can execute Python code at a time). That’s why we’re working on moving more of logic commonly used in the vision networks, as well as some others, to our C backends, so they can proceed without blocking other threads.
How to create model with sharing weight?

xiaozhun07: I want to create a model with sharing weights, for example: given two input A, B, the first 3 NN layers share the same weights, and the next 2 NN layers are for A, B respectively. How to create such model, and perform optimally?

apaszke: EDIT: we do support sharing Parameters between modules, but it’s recommended to decompose your model into many pieces that don’t share parameters if possible. We don’t support using the same Parameters in many modules. Just reuse the base for two inputs: <SCODE>class MyModel(nn.Module):
    def __init__(self):
        self.base = ...
        self.head_A = ...
        self.head_B = ...

    def forward(self, input1, input2):
        return self.head_A(self.base(input1)), self.head_B(self.base(input2))
        
<ECODE>

vladimir: in your example, what will happen to gradients of self.base? will they be calculated taking into account both input1 and input2?

apaszke: Yes, you can use the same module multiple times during forward.

jekbradbury: 

apaszke: Yeah they are supported, sorry for this. But it’s still considered better practice to not do it. I’ve updated the answer.

11189: In this code, he make share modules(‘G_block_share/D_block_share’) out of class, and then use these share modules in different two classes(‘Generator A&B or Discriminator A&B)… This code is right way to share weights between two generator/discriminators?

N_Hunter: Could you please tell us why it is better to not do it? Thanks

ZHANGHeng19931123: Dear Apaszke, thank you for your updates! But I am still a little confused about your answer. Like in your example, you have 3 modules (base, headA, headB),  but how could you decompose them into pieces that don’t share parameters? Looking forward to your answer, please! Thank you for your attention.

Baichuan: 

Baichuan: <SCODE>class MyModel(nn.Module):
    def __init__(self):
        self.base1 = ...
        self.base2=...
        self.head_A = ...
        self.head_B = ...

    def forward(self, input1, input2):
        return self.head_A(self.base1(input1)), self.head_B(self.base2(input2))
        
<ECODE>

Hazel: But in this case, how would base1 and base2 share the same weights? It seems like base1 + head_A and base2 + head_B are totally separate models.

dariodematties: Thanks!

Alex_Li: I think sometimes you have to use weight sharing, like in the case where you want one layer to be the transpose of another. For this case, one can do this: <SCODE>shared_w =  torch.rand((n_y, n_z))*.2 - .1 # initialize somehow
self.yzdecoding = nn.Linear(n_y, n_z) # Create shared layers
self.zydecoding = nn.Linear(n_z, n_y)
self.yzdecoding.weight = nn.Parameter(shared_w.T) # Share weights
self.zydecoding.weight = nn.Parameter(shared_w)
<ECODE> Note that the (n_y, n_z) Linear layer has weights of shape (n_z, n_y), which may not be intuitive at first.
Installation problem for python 3.6

Hamid: 

apaszke: What’s your OS? Do you have a 64-bit system?

Hamid: My OS is Ubuntu 14.04 64-bit

smth: <SCODE>wget https://s3.amazonaws.com/pytorch/whl/cu75/torch-0.1.8.post1-cp36-cp36m-linux_x86_64.whl
mv torch-0.1.8.post1-cp36-cp36m-linux_x86_64.whl torch-0.1.8.post1-cp36-none-linux_x86_64.whl
pip install torch-0.1.8.post1-cp36-none-linux_x86_64.whl
<ECODE> <SCODE>pip3 --version
python3 --version
<ECODE>

Hamid: 

Hamid: 

smth: yes, the pip needs to be linked to python3.6, because the wheel file is for python 3.6

Hamid: Could  you tell how can I do that?

Hamid: 

herlimenezes: Doesn’t it work with python 3.5?
Pre-trained network demo

Atcold: <SCODE># get input image
import skimage.io
import os
file_name = '26132.jpg'
if not os.access(file_name, os.R_OK):
    file_URL = 'http://www.zooclub.ru/attach/26000/26132.jpg'
    os.system('wget ' + file_URL)
img = skimage.io.imread(file_name)


# get model
import torchvision
resnet_18 = torchvision.models.resnet18(pretrained=True)


# get classes
file_name = 'synset_words.txt'
if not os.access(file_name, os.W_OK):
    synset_URL = 'https://github.com/szagoruyko/functional-zoo/raw/master/synset_words.txt'
    os.system('wget ' + synset_URL)
classes = list()
with open(file_name) as class_file:
    for line in class_file:
        classes.append(line.strip().split(' ', 1)[1].split(', ', 1)[0])
classes = tuple(classes)


# define image transformation
from torchvision import transforms as trn
centre_crop = trn.Compose([
        trn.ToPILImage(),
        trn.Scale(256),
        trn.CenterCrop(224),
        trn.ToTensor(),
        trn.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])


# get top 5 probabilities
from torch.autograd import Variable as V
from torch.nn import functional as f
x = V(centre_crop(img).unsqueeze(0), volatile=True)
logit = resnet_18.forward(x)
h_x = f.softmax(logit).data.squeeze()
probs, idx = h_x.sort(0, True)
for i in range(0, 5):
    print('{:.3f} -> {}'.format(probs[i], classes[idx[i]]))
<ECODE> And this is the output <SCODE>0.009 -> bucket
0.007 -> plunger
0.006 -> hook
0.005 -> water bottle
0.005 -> water jug
<ECODE> which should be, instead, roughly <SCODE>0.99 -> German shepherd
0.01 -> malinois
0.00 -> Norwegian elkhound
0.00 -> Leonberg
0.00 -> red wolf
<ECODE> And this is the input picture.

colesbury: 

Atcold: <SCODE>0.935 -> German shepherd
0.033 -> Leonberg
0.031 -> malinois
0.000 -> Norwegian elkhound
0.000 -> African hunting dog
<ECODE>

Atcold:
How to make custom method in nn.Module work with GPUs
Category: vision

moskomule: I’m trying to implement simple res-net like below and it works with CPU. <SCODE>class ResNet(nn.module):
      def __init__(self):
        super(Net, self).__init__()
        self.conv_1 = nn.Conv2d(3, 64, 4, stride=2)
        self.bn_1 = nn.BatchNorm2d(64)
        self.res_1 = self.__res_block(64, [32,32,128], True)
         ...
    def forward(self, x):
        x = self.conv_1(x)
        x = F.relu(self.bn_1(x))
        x = F.max_pool2d(x, 2, 2)
        x = self.res_1(x)
        ...

    def __res_block(self, in_channels,
                    nb_filters, right=False):
            def __res_base(_in_channels, out_channels,
                       kernel_size=1, padding=0):
            def g(x):
                x = nn.Conv2d(in_channels=_in_channels,
                              out_channels=out_channels,
                              kernel_size=kernel_size,
                              padding=padding)(x)
                x = nn.BatchNorm2d(num_features=out_channels)(x)
                return x
            return g

        def f(x):
            y = F.relu(__res_base(in_channels, nb_filters[0])(x))
            y = F.relu(__res_base(nb_filters[0], nb_filters[1],
                                  kernel_size=3, padding=1)(y))
            y = F.relu(__res_base(nb_filters[1], nb_filters[2])(y))
            if right is True:
                x = __res_base(in_channels, nb_filters[2])(x)
            return F.relu(x+y)
        return f
<ECODE> but it doesn’t work with GPU and throws TypeError, <SCODE>TypeError: _cudnn_convolution_full_forward received an invalid combination of arguments 
- got (torch.cuda.FloatTensor, torch.FloatTensor, torch.FloatTensor, torch.cuda.FloatTensor, tuple, tuple, int, bool), 
but expected (torch.cuda.RealTensor input, torch.cuda.RealTensor weight, torch.cuda.RealTensor bias, 
torch.cuda.RealTensor output, std::vector<int> pad, std::vector<int> stride, int groups, bool benchmark)
<ECODE>

albanD: Hi,

moskomule: 

alexis-jacq: 

albanD: Ho I see,

apaszke: 

moskomule: 

smth: 

moskomule: 

apaszke: 

marioviti:
PyTorch serialization

ablavatski1: Hi, Awesome library! I’d like to ask if it is possible to save a trained PyTorch model in ("*.t7") and read it in Torch. Thanks

apaszke: No, we don’t have such option. PyTorch allows for creating much more complex models than Lua Torch without a need to define a lot of helpers, and because of that it’s not easy to translate models in this direction.

ablavatski1: Thanks for the fast reply!
Tensors as items in multiprocessing.queue

florin: <SCODE>import torch
import torch.multiprocessing as mp

def put_in_q(idx, q):
    q.put(torch.IntTensor(2, 2).fill_(idx))
    # q.put(idx)  # works with int, float, str, np.ndarray, but not torch.Tensor

q = mp.Queue()

p = mp.Process(target=put_in_q, args=(0, q))
p.start()

x = q.get()
print(x)

p.join()
<ECODE> The error I get: <SCODE>Traceback (most recent call last):
  File "test_torch_queue.py", line 15, in <module>
    x = q.get()
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/queues.py", line 113, in get
    return _ForkingPickler.loads(res)
  File "/home/florin/Tools/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/reductions.py", line 72, in rebuild_storage_fd
    fd = df.detach()
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 57, in detach
    with _resource_sharer.get_connection(self._id) as conn:
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/resource_sharer.py", line 87, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/connection.py", line 493, in Client
    answer_challenge(c, authkey)
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/florin/Tools/anaconda3/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
<ECODE> Thanks!

apaszke: The subprocess needs to be alive at the time when the master process receives the Tensor. There are two ways to fix that: This example should work: <SCODE>import torch
import torch.multiprocessing as mp

def put_in_q(idx, q, evt):
    q.put(torch.IntTensor(2, 2).fill_(idx))
    evt.wait()

q = mp.Queue()

evt = mp.Event()
p = mp.Process(target=put_in_q, args=(0, q, evt))
p.start()

x = q.get()
evt.set()
print(x)

p.join()
<ECODE>

florin: Thank you for the quick reply!
Help debugging DenseNet model on CIFAR-10
Category: vision

bamos: Hi PyTorch community, -Brandon.

apaszke: One thing is that this: <SCODE>for param_group in optimizer.state_dict()['param_groups']:
<ECODE> should be replaced with that: <SCODE>for param_group in optimizer.param_groups:
<ECODE> I know that the first version appears in the ImageNet example, but it no longer works as expected. I can’t see anything wront at a glance, but I’ll try to look more carefully into it sometime.

bamos: Thanks for quickly looking! I fixed that. (It doesn’t sound like you expected this to help the convergence – the model’s still not converging as before.)

bamos: Thanks! My training code does successfully train a known model (VGGnet) on CIFAR-10, so my logic is also that there’s something wrong with my model. However I have also exactly compared my model’s outputs and gradients to the official model. I thought about trying to remove the passthrough connections from my DenseNet implementation to further debug this. However I haven’t tried this yet because I’ve never seen reports of such an architecture (even correctly implemented) converging on CIFAR-10. And if my implementation of this did converge, then it would indicate that there’s a problem of layers that concatenate the input and output. So to directly check if there’s a problem with this kind of operation, I used numdifftools to numerically check the gradients of a single PyTorch layer that concatenated the input to a fully-connected operation. As another idea of breaking the DenseNet into a known architecture, I could start with a ResNet architecture that’s known to converge and then start adding DenseNet features. However these intermediate architecture are not known to converge, so if it doesn’t work, then I won’t know if it’s because of a code bug or something more fundamental.

bamos: Adam’s been helping me debug this over Slack today and we’ve solved it! We found a new PyTorch bug with cudnn that comes up with DenseNet-style layers. After fixing this, my DenseNet model converges much better than before, and I’ll update my repo with the current results shortly. My understanding is that Adam will push a short patch to PyTorch master soon. Thanks again for the help, Adam. -Brandon.

apaszke: The fix is already in master.

ClementPinard: What was the problem ? Been having some convergence issues with my network which has some layers concatenation just as yours. My problems are not necessarily related to yours, but I’m glad you could figure it out with adam !

apaszke: The problem arised form concatenating outputs of convolution along second dimension. This lead to calling conv’s backward with non-contiguous gradients, and we were overly smart about reusing cuDNN descriptors, so the backend thought that it’s contiguous. The fix is to either disable cuDNN or rebuild pytorch.

fmassa:
Sum of matrices with different dimensions

lcelona: I am trying to sum two tensors with dimensions: 
a: 10 x 49 x 1024 
b: 10 x 1024 Thanks

apaszke: How do you want to add these matrices? They have different numbers of elements.

lcelona: Don’t hate me. In Tensorflow, the following code works: I’ll replace my code using the following: Correct?

apaszke: This should do it for you: <SCODE>c = a + b.unsqueeze(1).expand_as(a)
<ECODE>

apaszke: But your code would work too.

lcelona: Thank you for your help!
Runtime error occurs when using .cuda(1)

big_tree: Hi all, I try to use pytorch on the 2nd GPU, <SCODE>`a = torch.ones(1).cuda(1)
 b = torch.ones(1).cuda(1)
 c = torch.cat((a,b),0)`
<ECODE> Then an error comes out: RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.7_1485444530918/work/torch/lib/THC/generic/THCTensorCopy.c:65 How can I fix this?

big_tree: 

apaszke: 

big_tree: 

WarBean: Encounter the same problem. I have just updated to the latest version but the error sitll rises. Has it been fixed? If not, is there any workaround?

WarBean: For now, it seems that I can workaround by using GPU 0.

apaszke:
WhiteNoise Layer for DCGAN tutorial

lmoss: Hi everyone, Any hint would be welcome and I’m happy to make pull request as an added feature once done.

apaszke: 

lmoss: I will have another look at the tutorials, thank you in the meantime.

lmoss: I’m replying to myself, in case anyone else runs into this issue.

apaszke: You could also just sample white noise, wrap it in a Variable and add that. I think that would be a more elegant solution.

supakjk: 

apaszke: If you want to add the noise in the middle of a network, just use two sequentials.
Error on transpose and view

ypxie: <SCODE>import torch
import torch.nn.functional as F
def softmax(input, axis=1):
    """ 
    Apply softmax on input at certain axis.
    
    Parammeters:
    ----------
    input: Tensor (N*L or rank>2)
    axis: the axis to apply softmax
    
    Returns: Tensor with softmax applied on that dimension.
    """
    
    input_size = input.size()
    
    trans_input = input.transpose(axis, len(input_size)-1)
    trans_size = trans_input.size()

    input_2d = trans_input.view(-1, trans_size[-1])
    soft_max_2d = F.softmax(input_2d)
    
    soft_max_nd = soft_max_2d.view(*trans_size)
    
    return soft_max_nd.transpose(axis, len(input_size)-1)

aa= torch.randn(3,4,4)
print aa

soft_1 = softmax(aa, axis = 1)
print soft_1
<ECODE> gives the following error: <SCODE>File "/local/anaconda2/lib/python2.7/site-packages/torch/tensor.py", line 214, in view
    raise ValueError("input should be contiguous")
ValueError: input should be contiguous
<ECODE>

longcw: You can try this: <SCODE>input_2d = trans_input.contiguous().view(-1, trans_size[-1])
<ECODE>

apaszke:
PyTorch Resources
Category: Site Feedback

ritchieng: PyTorch is relatively new compared to other frameworks and I had issues finding more guides and tutorials.

Ismail_Elezi: Excellent. Will look into that and hopefully contribute in the future, although I am still using both PyTorch and Tensorflow for my projects.

ritchieng: In less than 1 week I’ve added more than 20 implementations and it’s by far the largest list currently out there on PyTorch. I will grow this list with at least an update every 3 days. I’m a proponent of reproducible research so the section on paper implementation would be rapidly growing.
PReLU and Conv3d bugs with 5d Tensors

abweiss: Here is an example of torch.nn.PReLU(num_parameters) acting on a 5d Tensor: <SCODE>out = nn.PReLU(8)(Variable(torch.rand(2,8,16,16,16)))
<ECODE> The error looks like: <SCODE>RuntimeError: wrong number of input planes at /data/users/soumith/miniconda2/conda-bld/pytorch-cuda80-0.1.8_1486040640754/work/torch/lib/THNN/generic/PReLU.c:49
<ECODE> Here is an example of torch.nn.Conv3d(bias=False) acting on a 5d Tensor: <SCODE>out = nn.Conv3d(8,16, kernel_size=3, padding=1, bias=False)(Variable(torch.rand(2,8,16,16,16)))
<ECODE> The error looks like: <SCODE>TypeError: FloatVolumetricConvolutionMM_updateOutput received an invalid combination of arguments - got (int, torch.FloatTensor, torch.FloatTensor, torch.FloatTensor, NoneType, torch.FloatTensor, int, int, int, int, int, int, int, int, int), but expected (int state, torch.FloatTensor input, torch.FloatTensor output, torch.FloatTensor weight, torch.FloatTensor bias, torch.FloatTensor finput, int kT, int kW, int kH, int dT, int dW, int dH, int pT, int pW, int pH)
<ECODE> Lua Torch had similar problems with 5d Tensors, so maybe this is a back end issue.

smth: 

apaszke:
Are only used parameters updated?

supakjk: When A is a module with multiple sub-modules, where only a sub-set of the sub-modules are used dependent on the input, for example, A is defined as follows, <SCODE>class A(nn.Module):
    def __init__(self):
        self.common = nn.Linear(100, 50)
        self.module1 = nn.Linear(50,30)
        self.module2 = nn.Linear(50,2)
    def forward(self, input, idx):
        commonOut = self.common(input)
        if idx == 0:
            return self.module1(commonOut)
        else:
            return self.module2(commonOut)

opt = torch.optim.Adam(A.parameters()) 
<ECODE> If it does, is there any way I can get the list of parameters that are actually updated? Thanks

apaszke: No, we don’t do that. There’s no way how the optimizer would know which parameters were or weren’t used, and there’s no possible way to find out about that. It would require us to impose some strict requirements, that don’t really make sense in most use cases, for only minor improvements in other. Why is it such a problem that these parameters are getting updated? Is your script slow? It shouldn’t add a lot of overhead.

supakjk: Thanks!

apaszke: 

jablongo: Aside from performance considerations, if the error is only being evaluated in terms of the output of a single module, aren’t there other (possibly negative) implications of updating the weights for components unused in the forward pass?
Speed benchmark on VGG16
Category: vision

tdeboissiere: I am testing pytorch’s speed on a simple VGG16 benchmark and I have noticed the following timings: Do these timings sound reasonable and is there a reason why Iteration 1 is much faster than the rest ? System specs: Thanks !

apaszke: Also, you’re measuring the time of CUDA copies, but using pinned memory and async transfers would be faster than the simplest approach.

tdeboissiere: Thanks! Can you point me to the documentation to re-implement the benchmark with pinned memory and async transfers ?

apaszke: