foldl/torch-forum huggingface dataset:
How to show a image in jupyter notebook with pytorch easily?

richardhahahaha: like in itorch, I can use itorch.image to show a image in the notebook.

amdegroot: Yeah this frustrated me a lot too because it’s so easy with itorch… You have a few options with python but there’s not a stand-alone command that I’m aware of for displaying an image from a PyTorch Tensor.  If you convert to a PIL image then you can just execute the Image variable in a cell and it will display the image.  To load to PIL: <SCODE>img =  Image.open('path-to-image-file').convert('RGB')
<ECODE> Or to convert straight from a PyTorch Tensor: <SCODE>to_pil = torchvision.transforms.ToPILImage()
img = to_pil(your-tensor)

<ECODE>

chenyuntc: this is how I display an image: <SCODE>import torch as t
from torchvision.transforms import ToPILImage
from IPython.display import Image
to_img = ToPILImage()

# display tensor
a = t.Tensor(3, 64, 64).normal_()
to_img(a)

# display imagefile
Image('/path/to/my.png')
<ECODE>

smth: you can look at my notebook here, but yea seems like no standard solution. Basically, I do: <SCODE>%matplotlib inline
def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1,2,0)), interpolation='nearest')
<ECODE>

oya163: Hi Soumith, I would like to know how I can use your solution, if I am using iterator like this <SCODE>dataiter = iter(trainloader)
images, labels = dataiter.next()
<ECODE> Thank you in advance.

farrokhi: I have same question…

Vimos: <SCODE>    display(Image(image_path))
<ECODE>

talha3111997: This is what I did, hope can be helpful for someone. <SCODE>from PIL import Image
`image = Image.open('img_path.jpg').convert('RGB')
 plt.imshow(np.transpose(np.asarray(content.squeeze()),(1,2,0)))`

<ECODE>

xiaoyanzhuo: What I did: transfer the tensor to numpy array, reshape(if needed), and use plt.imshow(): <SCODE>import matplotlib.pyplot as plt
img_np_arr = img_tensor.numpy()   # transfer the pytorch tensor(img_tensor) to numpy array
img_np_arr.shape    # check shape before reshape if needed
img_np_arr_reshaped = img_np_arr.reshape(img_w, img_h)  # reshape to 2-dims image
plt.imshow(img_np_arr_reshaped, cmap='gray')   # when display the grayscale image
<ECODE>

ramapinnimty: This worked flawlessly for displaying an image from CIFAR-100 dataset. Thank you!
How to index a two-dimension tensor with two single-dimension tensors

Coldmooon: I want to index a two-dimension tensor like I do in numpy. <SCODE>TypeError: indexing a tensor with an object of type LongTensor. The only supported types are integers, slices, numpy scalars and torch.LongTensor or torch.ByteTensor as the only argument.
<ECODE> How do I index a two-dimension tensor using other two tensors?

smth: a[b, :][:, b] We plan to tackle this soon.

Coldmooon: Thanks for your reply. But this still gives the same error: <SCODE>>>> a[b,:][:,b]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: indexing a tensor with an object of type LongTensor. The only supported types are integers, slices, numpy scalars and torch.LongTensor or torch.ByteTensor as the only argument.
<ECODE> My pytorch vision is: <SCODE>$ conda list | grep torch  
pytorch                   0.1.10               py35_1cu80  [cuda80]  soumith
torchvision               0.1.6                   py35_19    soumith
<ECODE>

b64406620: 

YankuHorn: 

dblythe: x[b].transpose(1, 0)[b].transpose(1, 0) should do the trick. Which seems self-contradictory

dblythe: Can also try adding something like: <SCODE>_oldgetitem = torch.FloatTensor.__getitem__


def _getitem(self, slice_):
    if type(slice_) is tuple and torch.LongTensor in [type(x) for x in slice_]:
        i = [j for j, ix in enumerate(slice_)
             if type(ix) == torch.LongTensor][0]
        return self.transpose(0, i)[slice_[i]].transpose(i, 0)
    else:
        return _oldgetitem(self, slice_ )


torch.FloatTensor.__getitem__ = _getitem
<ECODE>
Run two different jobs in parallel on same GPUs, I got my GPU locked up

hbkunn: firstly I’ve training on a regular python file using 4 GPUs using function Dataparallel. Then I loaded saved parameters using ipython notebook through SSH while the previous job is still running. when I load it on a single GPU instead of Dataparallel, it shows that weight doesn’t exist.So instead I use Dataparallel function on the same GPUs just like training process, then the problem occurred. The ipython froze, and I immediately kill the job. Then My GPU is locked up like the picture shows. I can restart any job but it only shows 1MB memory whatever I tried. I’ve ran into the same problem before, reboot can do but my peers are also using this remote server. What can I do ;-(, I searched through ‘ps -ef’ but still cannot find relevant jobs that caused the problem. What have I done ;-(.

apaszke: So the problem is because the NVIDIA libraries we’re using for inter-GPU communication in DataParallel do some funky stuff and they can leave the driver in some inconsistent state. Just remember to never launch multiple DataParallel jobs that share some of the GPUs (it’s ok to run one job on GPU 0, 1 and anoher on 2, 3).

hbkunn: I have rebooted my server, but still some strange errors occur occasionally like ‘RuntimeError: cuda runtime error (4) : unspecified launch failure at /home/soumith/local/builder/wheel/pytorch-src/torch/lib/THC/generic/THCTensorCopy.c:18’. And when I ran ‘nvidia-smi’ sometimes it’s extremely slow…

hbkunn: e… it seems like a serious problem. unspecified launch failure constantly occurs about 1 or 2 hours after I launch my code. I can’t find any solution related. Should I reinstall my NVIDIA driver…?

smth: is your GPU becoming too hot? occurs after 1 to 2 hours of launching your code sounds like that might be a problem? (because Unspecified Launch Failure might sometimes be that)

yzhu: maybe try a complete power cycle? shut down the machine for a few seconds?

cjmcmurtrie: Not sure why but this comment was addressed to me? I received an email notification.

hbkunn: 

hbkunn:
Is it possible to make a function transparent between cpu and gpu?

david-leon: I defined a function in which there’re temporary variables are defined, for example: <SCODE>def foo(input_variable):    
    tmp = Variable(torch.zeros(input_variable.size()))    
    return torch.cat((input_variable, tmp), 1)       
<ECODE>

chenyuntc: by the way, you can use below to format your code it will be formated like this <SCODE>def foo(input_variable): 
     tmp = Variable(torch.zeros(input_variable.size())) 
     return torch.cat((input_variable, tmp), 1)
<ECODE> it’s  markdown sytanx

david-leon: Thanks, this really helps!

pranav: The following works when X is on CPU but fails to if its not: <SCODE>>>> X = Variable(torch.zeros(2, 2))
>>> torch.Tensor.new(X.data, torch.zeros(2, 2))

 0  0
 0  0
[torch.FloatTensor of size 2x2]

>>> X = Variable(torch.zeros(2, 2)).cuda()
>>> torch.Tensor.new(X.data, torch.zeros(2, 2))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-44093d4f4ab5> in <module>()
----> 1 torch.Tensor.new(X.data, torch.zeros(2, 2))

TypeError: unbound method new() must be called with FloatTensor instance as first argument (got FloatTensor instance instead)

<ECODE>

albanD: Hi,

david-leon: For batch size = 128, on my model way 1 runs 3 times slower than way 2 & 3 (~4s vs ~1.6s). Way3 is slightly faster than way 2. Is this supposed to be or am I doing wrong? Besides, is there any way to get the device a tensor currently resides on?

smth: what you want to do is: noise = x.data.new(x.size()).normal_() That will be the fastest.

pranav: 

david-leon: 

chenyuntc: do <SCODE>noise =x.data.new(*x.size()).normal_()
<ECODE>
What are items inside `torch.__builtins__` for?

dl4daniel: 

smth: these are python built-in objects / members.

dl4daniel: Given they are called builtin functions or classes or types, please correct me if I am wrong: <SCODE>any, __name__, StopAsyncIteration, RuntimeWarning, BufferError, NotImplementedError, UnboundLocalError, repr, UnicodeTranslateError, UnicodeWarning, max, IndentationError, BytesWarning, None, IsADirectoryError, __IPYTHON__, AttributeError, reversed, ConnectionError, BaseException, issubclass, ConnectionResetError, range, staticmethod, PendingDeprecationWarning, __doc__, AssertionError, GeneratorExit, chr, abs, RecursionError, MemoryError, __package__, frozenset, str, enumerate, ImportError, dreload, bytes, float, map, TabError, True, ZeroDivisionError, False, memoryview, super, type, filter, SystemError, getattr, callable, __build_class__, setattr, sum, round, ConnectionAbortedError, OSError, ValueError, dict, DeprecationWarning, RuntimeError, TypeError, pow, hasattr, exec, zip, ArithmeticError, help, LookupError, UserWarning, PermissionError, print, ReferenceError, SystemExit, IndexError, complex, all, license, UnicodeError, input, vars, open, KeyboardInterrupt, bytearray, StopIteration, ConnectionRefusedError, __import__, SyntaxError, KeyError, UnicodeEncodeError, dir, ProcessLookupError, sorted, FutureWarning, ascii, ChildProcessError, next, IOError, EnvironmentError, format, id, __spec__, ImportWarning, divmod, TimeoutError, EOFError, Ellipsis, OverflowError, object, eval, hash, locals, InterruptedError, delattr, int, FloatingPointError, credits, Exception, min, BlockingIOError, __loader__, globals, NotADirectoryError, NameError, slice, get_ipython, classmethod, UnicodeDecodeError, copyright, compile, bool, isinstance, ord, __debug__, len, hex, ResourceWarning, BrokenPipeError, Warning, tuple, oct, iter, FileNotFoundError, NotImplemented, set, property, SyntaxWarning, FileExistsError, bin, list
<ECODE>

jekbradbury:
Initialize different convolution layers of a network with different method

Soniya: I want to initialize different convolution layers of a network with different methods. Is there any way to do this thing? Please help me.

shaun: I found a course on Deep Learning from Chinese University of Hong Kong useful. They have a tutorial on PyTorch that shows how we can initialize different layers. 
Here 83 is the link to the course. 
Here 77 is the link to the slides for the tutorial. 
Here 63 is the link to the code that is discussed during the tutorial. 
Here 11 is the link to the audio for the class session. Slide number 14 talks about how to initialize parameters. In the audio at 36 minutes in, the instructor talks about this slide. I think this will help.
The training speed of between GPU and CPU is same?

xiaoxing_zeng: When I using Pytorch for conditional gan , I found that the training speed had little change no matter using GPU nor CPU. There are some mistake in my codes?

smth: 

Ismail_Elezi: Are you putting the net/model in cuda? Unlike Tensorflow, here you have to specify that you want to use GPU. For small models there isn’t much difference, but for big models there difference is quite big.

xiaoxing_zeng: 

Ismail_Elezi: Yes, you should send all the variables in cuda, but in addition you also need to send both the neural networks in cuda, in addition to the criterior (which contains the cost function). Something like: <SCODE>if opt.cuda:
    discriminator.cuda()
    generator.cuda()
    criterion.cuda()
    input, label = input.cuda(), label.cuda()
<ECODE> might do the trick.
Question about programming preference

LiZeng: when define a new net I find that I would use an object of a nn.Module class  for multiple times so I just defined this object one time and use it in every nn.Module as follow: <SCODE>feature = FeatureExtracter(SUBMODEL)
        
class Net1(nn.Module): 
        def forward(self,x):
            s_feats =feature(x,LAYER)
            ......#do something
            return ......

class Net2(nn.Module): 
        def forward(self,x):
            s_feats =feature(x,LAYER)
            ......#do something
            return ......
<ECODE> was that a bad programming habit ? or is there any reason in pytorch I shouldn’t do that, and instead define new object in every Net like this: <SCODE>class Net1(nn.Module): 
        def __init__(self):
            super().__init__()
            self.feature = FeatureExtracter(SUBMODEL)
        def forward(self,x):
            s_feats = self.feature(x,LAYER)
            ......
            return ......

class Net2(nn.Module): 
        def __init__(self):
            super().__init__()
            self.feature = FeatureExtracter(SUBMODEL)
        def forward(self,x):
            s_feats = self.feature(x,LAYER)
            ......
            return ......<ECODE>

smth: Both models are fine, neither is bad programming habit.
Backprop Through sparse_dense_matmul

elanmart: Do You have a roadmap for the whole sparse module? I’m curious if you’re planning to use cuSPARSE and if there is any way to contribute to this module. I’d need a backward pass for sparse_mat @ dense_mat. What are my options? Can I write a Function using the available spmm for forward, and add some quick-and-dirty cython code to do the backward? This would mean returning a sparse grad, is this supported?

ebetica: 

martinraison: Hi, for now sparse tensor support basically evolves with the needs of our respective projects. Basic GPU support is being worked on - it will rely on cuSPARSE for some operations. returning sparse gradients from backward should work in addition to the thread @ebetica pointed to, since pytorch supports hybrid tensors (i.e. tensors with both sparse and dense dimensions), we may add a sparse * dense -> hybrid function in the future (where “hybrid” here means one sparse dimension and one dense dimension). That would probably be more efficient in this case.

elanmart:
Variable logical operators bug?

Nadav_Bhonker: Consider the following code: <SCODE>from torch.autograd import Variable
t = torch.Tensor([2])
bool(torch.max(t) < 2)
Out[4]: 
False
bool(torch.max(t) < 3)
Out[5]: 
True
<ECODE> However, If you do the same with a Variable: <SCODE>bool(torch.max(v) < 2)
Out[6]: 
True
bool(torch.max(v) < 3)
Out[7]: 
True
<ECODE>

albanD: 

Nadav_Bhonker: Well that makes sense. Thanks.
Factorizarion machines

Gidi_Sh: 

smth: it is definitely not done yet (I am trying to keep tabs on user-implemented pytorch repositories)

Gidi_Sh: 

mzaradzki: Hi there, I recently started using PyTorch (more familiar with Keras) and here is 1st my attempt at Factorization Machine: Remark ; I implemented it using the O(k.N) formulation found in Steffen Rendle paper

Jack_Hessel: If you’re interested, I have also implemented a factorization machine in pytorch – I cythonized the forward and backward passes, and so it’s relatively fast. Definitely a work-in-progress still!
How can I convert the result from a Conv layer to a vector

rogetrullo: <SCODE>class Discriminator(nn.Module):
    def __init__(self, nc,ndf):
        super(Discriminator, self).__init__()
        self.features = nn.Sequential(
            # input is (nc) x 64 x 64
            nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf) x 32 x 32
            nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 2),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*2) x 16 x 16
            nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 4),
            nn.LeakyReLU(0.2, inplace=True),
            # state size. (ndf*4) x 8 x 8
            nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ndf * 8),
            nn.LeakyReLU(0.2, inplace=True)
            # state size. (ndf*8) x 4 x 4
            #nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
            #nn.Linear(ndf * 8,1),
            #nn.Sigmoid()
        )
        self.classifier= nn.Sequential(
            nn.Linear(self.inpelts,1),
            nn.Sigmoid()
            )
    def forward(self, input):
        x=self.features(input)        
        x = x.view(x.size(0), -1)#B,chans*H*W
        self.inpelts=x.size(1)
        print 'x shape ',x.size()
        output=self.classifier(x)
        return output
<ECODE>

smth: <SCODE>linear_weight = nn.Parameter(torch.randn(1, 1), requires_grad=True) # resize this in the forward function
<ECODE> And in the forward function: <SCODE># some stuff before, where x is the input to linear
if self.linear_weight.data.nelement() == 1:
   self.linear.weight.data.resize_(1, x.size(1)) # output_features x input_features)
   # initialize weights randomly
   self.linear.weight.data.normal_(-0.1, 0.1)
x = nn.functional.linear(x, self.linear_weight)
<ECODE>
Pixelwise weights for MSELoss

abweiss: I would like to use a weighted MSELoss function for image-to-image training.  I want to specify a weight for each pixel in the target.  Is there a quick/hacky way to do this, or do I need to write my own MSE loss function from scratch?

smth: you can do this: <SCODE>def weighted_mse_loss(input, target, weights):
    out = input - target
    out = out * weights.expand_as(out)
    # expand_as because weights are prob not defined for mini-batch
   loss = out.sum(0) # or sum over whatever dimensions
   return loss
<ECODE>

abweiss: Oh, I see.  There’s no magic to the loss functions.  You just calculate whatever loss you want using predefined Torch functions and then call backward on the loss.  That’s super easy.  Thanks!

liygcheng: what if I want L2 Loss ? just do it like this? <SCODE>def weighted_mse_loss(input,target,weights):
    out = (input-target)**2
    out = out * weights.expand_as(out)
    loss = out.sum(0) # or sum over whatever dimensions
    return loss

<ECODE> right ?

smth: 

Jordan_Campbell: 

delhi_loafer: 

JindongJiang: Same question, what is your thoughts now ?

delhi_loafer: 

JindongJiang: Thanks. Actually, variablization must be done if we need them operate at gpu(s).

picolo: 

ptrblck: 

picolo: I am trying an unsupervised approach to identify “important” areas of an image without explicitly pre-defining the weight-map. For the other question, I am not sure yet. I am thinking of a simple form like this - LW+(1/W) where L W is elementwise multiplication of learned weights and the MSE/MAE loss. and the second term prevents going too close to 0?
How can I do element-wise batch matrix multiplication?

wasiahmad: I have two tensors of shape (16, 300) and (16, 300) where 16 is the batch size and 300 is some representation vector. I want to compute the element-wise batch matrix multiplication to produce a matrix (2d tensor) whose dimension will be (16, 300). So, in short I want to do 16 element-wise multiplication of two 1d-tensors. I can do this using a for loop but is there any way, I can do it using torch API?

jekbradbury: 

wasiahmad: 

smth: What is batch element-wise multiplication? Can you show me your for loop?

ZeweiChu: If you have tensor a and b both of shape (16, 300) and you want to get a tensor of shape (16, 300) by element-wise multiplication, I assume you can do a*b?

wasiahmad: 

apaszke: 

wangg12: 

jekbradbury: 

wangg12: 

smth: we dont have auto-broadcasting yet.

dpappas: Could we use just if A is matrix and v is a vector ? It worked for me but i am not sure if the BP works fine.

wandering007: 

e.pignatelli: I solved like this, but maybe I am late to the party: <SCODE>def bmul(vec, mat, axis=0):
    mat = mat.transpose(axis, -1)
    return (mat * vec.expand_as(mat)).transpose(axis, -1)
<ECODE>
Shared parameter model and submodel BP

Marvin: <SCODE>class mm(nn.Module):
    def __init__(self):
        super(mm, self).__init__()
        self.m = nn.Linear(3,2)
        self.m2 = nn.Linear(2,4)
    def forward(self, input):
        o1 = self.m(input)
        o2 = self.m2(o1)
        return o1, o2

mmodel = mm()
optimizer = optim.SGD(mmodel.parameters(), lr=1, momentum=0.8)
#optimizer = optim.Adam(mmodel.parameters(), lr=1, betas=(0.5, 0.999))
input = Variable(torch.randn(1,3))
print(mmodel.m.weight.data)
print(mmodel.m2.weight.data)
mmodel.zero_grad()
output1, output = mmodel(input)
output1.backward(torch.ones(1,2))
#output.backward(torch.ones(1,4))
optimizer.step()
print(mmodel.m.weight.data)
print(mmodel.m2.weight.data)
<ECODE>
Convert pixel wise class tensor to image segmentation

bodokaiser: So as the label has format batch_size x 1 x height x width I can calculate cross entropy loss with: <SCODE>criterion = nn.CrossEntropyLoss()

# shape: batch_size x 1 x height x width = 1 x 1 x 256 x 256
inputs = autograd.Variable(images)
# shape: batch_size x 1 x height x width = 1 x 1 x 256 x 256
targets = autograd.Variable(labels)
# shape: batch_size x 1 x height x width = 1 x 256 x 256 x 256
outputs = model(inputs)

optimizer.zero_grad()
loss = criterion(outputs.view(-1, 256), targets.view(-1))
loss.backward()
optimizer.step()
<ECODE> I know that outputs[0][0][i][j| corresponds to the probability that the pixel at (i, j) belongs to class 1. So if want to transform outputs of shape 1 x 256 x 256 x 256 to 1 x 1 x 256 x 256 I would need to find the maximum (probability) of every pixel and assign it to the corresponding class value. I could do this manually by iterating over every class and pixel with numpy but I wonder if there is any better way using tensor operations?

fmassa: <SCODE>nllcrit = nn.NLLLoss2d(size_average=True) # need to write a functional interface for it
def criterion(input, target):
    return nllcrit(F.log_softmax(input), target)
<ECODE> <SCODE>output = output.permute(0, 2, 3, 1).view(-1, ncls).squeeze()
target = target.view(-1).squeeze()
<ECODE>

bodokaiser: 

jekbradbury: In torch, max and argmax are computed together and returned as a tuple. So outputs.max(0)[1] is the native torch way to do this.

bodokaiser: Any ideas what could be wrong? This happens with both models (UNet and 1-layer conv) so I guess there must be something wrong with loss or optimization.

meetshah1995: <SCODE>            # outputs.shape =(batch_size, n_classes, img_cols, img_rows) 
            outputs = outputs.permute(0, 2, 3, 1)
            # outputs.shape =(batch_size, img_cols, img_rows, n_classes) 
            outputs = outputs.resize(batch_size*img_cols*img_rows, n_classes)
            labels = labels.resize(batch_size*img_cols*img_rows)
            loss = F.cross_entropy(outputs, labels)
<ECODE>

fmassa:
Batch processing of sequences with different length

octopusyo: Hello, I have came up with something like this. q1 and q2 are just list of integer word indices. <SCODE> def SimilarityModule(nn.Module)
     def __init__():
         self.embedding = nn.Embedding(...)
     ....
    def forward(self, q1, q2):
        q1_embeds = self.embeddings(q1)
        q2_embeds = self.embeddings(q2)
        #mean over whole sequence
        q1_repre = torch.mean( q1_embeds, 0 ) 
        q2_repre = torch.mean( q2_embeds, 0 )

        dot_product = torch.dot( q1_repre, q2_repre ) / torch.norm( q1_repre, 2 ) / torch.norm( q1_repre, 2 )

        return dot_product
<ECODE> I have a question how properly create a batch of the sequences with variable length and feed it to the module? Thank you in advance for help!

smth: 

octopusyo: Thank you , Is there any way to work with “truly” variable length sequences? For instance, I would want to have a CNN with a RNN on top, I would also need masking?

jekbradbury: You would use padding in the CNN, then take the output and pass it to pack_padded_sequence for the RNN.

octopusyo: 

jekbradbury: Masking is mandatory but the PyTorch RNNs natively support variable length sequences (created by pack_padded_sequence) and will correctly avoid processing the padding tokens, even for the reverse direction of a bidirectional RNN.

aradhyamathur: Could you plz point to some example where variable length sequences are being passed in a batch

josmi9966: I do not understand what is really meant by “masking” - could anyone explain please?

flyaway: