akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to create zero dimensional tensor?

maris_ancans: <p>Say I have some float number, how would I create a tensor from only this one number?<br>
Simple tensor constructor would be nice, like Tensor(45) or even initializer list Tensor({23, 5, 64})</p>

tom: <p>Does <code>torch::tensor(1.23)</code> not work?</p>
<p>Best regards</p>
<p>Thomas</p>

maris_ancans: <p>oh, ok, I see what I did wrong. I had torch::Tensor(1.23) not torch::tensor(1.23). This is a bit confusing. I guess I got this from python, because in python it was possible to write FloatTensor(1.23).<br>
Thank you</p>

tom: <p>In Python PyTorch, <code>torch.tensor</code> has been <a href="https://pytorch.org/blog/pytorch-0_4_0-migration-guide/#creating-tensors" rel="nofollow noopener">the recommended way</a> for a while.</p>
<p>Best regards</p>
<p>Thomas</p>
Ordering of elements in pytorch module's backward hook?

lakshya97: <p>Hi,</p>
<p>I’ve been trying to use the hook functionality in pytorch and have run into a few questions about it.</p>
<p>To begin with, here are both the signature and docstring for the register_backward_hook function<br>
“”"</p>
<pre><code>def register_backward_hook(self, hook): 
    The hook will be called every time the gradients with respect to module
    inputs are computed. The hook should have the following signature::

       hook(module, grad_input, grad_output) -&gt; Tensor or None

    The :attr:`grad_input` and :attr:`grad_output` may be tuples if the
    module has multiple inputs or outputs. The hook should not modify its
    arguments, but it can optionally return a new gradient with respect to
    input that will be used in place of :attr:`grad_input` in subsequent
    computations.

    Returns:
        :class:`torch.utils.hooks.RemovableHandle`:
            a handle that can be used to remove the added hook by calling
            ``handle.remove()``

    .. warning ::

        The current implementation will not have the presented behavior
        for complex :class:`Module` that perform many operations.
        In some failure cases, :attr:`grad_input` and :attr:`grad_output` will only
        contain the gradients for a subset of the inputs and outputs.
        For such :class:`Module`, you should use :func:`torch.Tensor.register_hook`
        directly on a specific input or output to get the required gradients.

    """
</code></pre>
<p>As stated, the expected signature for a hook function (for a module) is hook(module, grad_input, grad_output). grad_input and grad_output appear to be tuples whenever a node has multiple inputs.</p>
<p>My question is this: what is the structure of the tuple of elements in grad_input? For a basic MNIST classification net that I’ve set up, it appears to be: (grad_input_to_layer, grad_weights, grad_bias) respectively, where grad_input_to_layer is the gradients with respect to the input of the layer, grad_weights is the gradient with respect to the weights of the layer, and grad_bias is the gradient with respect to the layer’s bias.</p>
<p>Is there a specific guarantee for what the structure of the grad_input and grad_output tuples are?</p>

tom: <p>The module backward <a href="https://discuss.pytorch.org/t/should-backward-hook-be-registered-before-forward-pass/28798/4">isn’t really what you expect</a>.</p>
<p>Best regards</p>
<p>Thomas</p>

lakshya97: <p>Thanks Tom. Do you have a recommendation as to how to proceed then for our use case? I would simply like to figure out what is being logged and in what order, but the link seems to suggest the hook is “broken”?</p>

tom: <p>Yeah, well the hook is broken. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"><br>
But this in particular means that the structure just comes from the last operation in the forward of the module.<br>
For some modules (conv, linear that doesn’t use broadcasting), that is OK, for others not so much.</p>
<p>Best regards</p>
<p>Thomas</p>
Irfft onesided=True does not merge the doubled centerline

xvdp: <pre><code class="lang-python">z = torch.zeros([1,3,224,224])
fft = torch.rfft(z, 2, onesided=True)
out = torch.irfft(fft, 2, onesided=True)
print(z.shape, out.shape)
#torch.Size([1, 3, 224, 224]) torch.Size([1, 3, 224, 225])
</code></pre>
<p>I understand that rfft duplicates centerline, irfft should remove it, i suppose. Im using onesided=False instead, but unless im mistaken theres a few code lines that must be missing<br>
torch.<strong>version</strong> ‘1.1.0’</p>

SimonW: <p><a href="https://pytorch.org/docs/stable/torch.html#torch.irfft" class="onebox" target="_blank" rel="noopener">https://pytorch.org/docs/stable/torch.html#torch.irfft</a></p>
<blockquote>
<p>Due to the conjugate symmetry,  <code>input</code>  do not need to contain the full complex frequency values. Roughly half of the values will be sufficient, as is the case when  <code>input</code>  is given by <a href="https://pytorch.org/docs/stable/torch.html#torch.rfft"> <code>rfft()</code> </a> with  <code>rfft(signal, onesided=True)</code> . In such case, set the  <code>onesided</code>  argument of this method to  <code>True</code> . Moreover, the original signal shape information can sometimes be lost, optionally set  <code>signal_sizes</code>  to be the size of the original signal (without the batch dimensions if in batched mode) to recover it with correct shape.</p>
</blockquote>

xvdp: <aside class="quote no-group" data-username="SimonW" data-post="2" data-topic="50534">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/simonw/48/1702_2.png" class="avatar"> SimonW:</div>
<blockquote>
<p>signal_sizes</p>
</blockquote>
</aside>
<p>Thank you, I had missed the ‘signal_sizes’ argument. much appreciated</p>
How to get gradient of hidden state of LSTM?

Ericcc: <p>Hi, I have used Pytorch for some time. But when I read the example code about <a href="https://github.com/pytorch/examples/blob/master/word_language_model/main.py" rel="nofollow noopener">language model</a>, I’m quite confused about how to get gradient of hidden state.</p>
<pre><code class="lang-auto">    hidden = model.init_hidden(args.batch_size)
    for batch, i in enumerate(range(0, train_data.size(0) - 1, args.bptt)):
        data, targets = get_batch(train_data, i)
        # Starting each batch, we detach the hidden state from how it was previously produced.
        # If we didn't, the model would try backpropagating all the way to start of the dataset.
        hidden = repackage_hidden(hidden)
        model.zero_grad()
        output, hidden = model(data, hidden)
        loss = criterion(output.view(-1, ntokens), targets)
        loss.backward()
</code></pre>
<p>But when I print <strong>hidden[0].grad</strong> or <strong>hidden[1].grad</strong> after <strong>loss.backward()</strong>, I got <strong>None</strong>. I have tried two approaches to get gradient.</p>
<pre><code class="lang-auto">hidden[0].register_hook(get_gradient)
hidden[1].register_hook(get_gradient)
</code></pre>
<p>And</p>
<pre><code class="lang-auto">hidden[0].retain_grad()
hidden[1].retain_grad()
</code></pre>
<p>However, neither way works. So how can get gradient of hidden state and ceil state?</p>

tom: <p>There are two things:</p>
<ul>
<li>The variable hidden is overwritten by <code>output, hidden = model(data, hidden)</code>, so if you want the initial hidden state, you would have to instead do <code>output, hidden_new = ...</code>.</li>
<li>The initial hidden state likely does not require grad, so you need <code>hidden[0].requires_grad_()</code> for the initial hidden state (for the final hidden state, <code>new_hidden[0].retain_grad_()</code> is good.</li>
</ul>
<p>All this assumes you are indeed using the LSTM and hidden and new_hidden are tuples.</p>
<p>Best regards</p>
<p>Thomas</p>
Distinguish forward function between test and train phase

Kamil_Adamczewski: <p>In the forward function, sometimes one may want to perform an action only at the test phase, say. Is there any built-in variable that allows to distinguish whether forward function was called during test or train phase?</p>

ptrblck: <p>You could use the internal <code>self.training</code> attribute to check the current state.</p>

Kamil_Adamczewski: <p>This boolean variable is exactly it! Thank you!</p>
Dropout for long tensor?

david-leon: <p>Currently it prompts <code>"fused_dropout" not implemented for 'Long'</code>, whereas it’s not a problem for Theano.</p>
<p>Is this feature we should wait for a while? or is it a bug?</p>

tom: <p>Neither.<br>
The problem is that modern dropout scales the outputs by 1/p. This would mean that you can only use 1/n, n integer . Also you don’t get autograd with long tensors. As such, the use-case seems too limited to include it in PyTorch given that <code>torch.empty_like(a).bernoulli_(keep_prob) * a</code> will give you an (unscaled) dropout for long with not much code.</p>
<p>Best regards</p>
<p>Thomas</p>

david-leon: <p>There’s no much sense to do mandatory scaling in Dropout, this feature should be optional controlled by a input argument.</p>
<p>Anyway, thanks for replying.</p>

ptrblck: <aside class="quote no-group" data-username="david-leon" data-post="3" data-topic="50914">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/david-leon/48/231_2.png" class="avatar"> david-leon:</div>
<blockquote>
<p>There’s no much sense to do mandatory scaling in Dropout, this feature should be optional controlled by a input argument.</p>
</blockquote>
</aside>
<p>I’m not so sure about this claim, as the expected values will be off, which will most likely result in bad validation and test performance as described in the <a href="http://jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf">Dropout paper</a>.</p>
Libtorch support for PSROI Pooling and PSROI Align layer?

jinfagang: <p>Hi, I am currently try using light head rcnn, I got it trained in pytorch.<br>
But I have some questions about this:</p>
<ol>
<li>
<p>how to export this model which using self-defined layer into libtorch traced model?</p>
</li>
<li>
<p>How to convert the model to onnx and then convert it into trt engine??</p>
</li>
</ol>
<p>Does any one worked with this before?</p>
Is there any example about how to deploy PyTorch Model in Production (C++)?

Yin_Zheng: <p>Is there any example about how to deploy a pytorch model in production environment (C++), for PyTorch 1.0 version?</p>

tom: <p>The <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">C++ export tutorial</a> shows you how to export your model and load and call it model from C++.<br>
How you write the surrounding application would likely depend on your use case.</p>
<p>Best regards</p>
<p>Thomas</p>

Yin_Zheng: <p>Thanks very much, Tom!</p>
Optimize Training Speed for Recommendation System Using Subnetworks

Misael_Manjarres: <p>I am building a recommendation system inspired by YouTube’s “Deep Neural Networks for YouTube Recommendations” paper. I will need to execute recommendations in real time so I structured it with low latency predictions in mind. The structure is the following</p>
<pre><code class="lang-auto">      |User Features|                         |Item Features|
               |                                   |
|Fully Connected NN_user|              |Fully Connected NN_item|
                \                        /
              |Concatenated output of both NNs|
                              |
                     |Fully Connected NN|
                              |
                           |output|
</code></pre>
<p>This is all one network built using two sub-networks.</p>
<p>The reason I did it this way is to create rich embeddings for the user and item based on their features which I could then store. At prediction time, I can retrieve the stored embeddings, then only the top NN needs to be executed and is therefore very fast. In testing, the model gives good results.</p>
<p>My question is about decreasing the time it takes to train this model. Is there a way for Pytorch to execute the sub-networks in parallel? Using DataParallel splits that data and trains it in parallel, but I think that the two sub-NN are trained one after the other, even though they don’t need to be. The forward section of the model has the following structure:</p>
<pre><code class="lang-auto">def sub-network(features, **params):
      ....

def forward(user_features, item_features):
      user_embedding = sub-network(user_features)
      item_embedding = sub-network(item_features)
      x = torch.cat([user_embedding, item_embedding],1)
      ...
</code></pre>
<p>What is a good strategy for parallelizing the execution of the sub-network functions?</p>

tom: <p>You do have several GPUs to make this worthwile, right?<br>
Given the asynchronous nature of of GPU computation, you can just move one network and inputs to the second GPU. Then it will be queued serially, but executed in parallel. Just be sure to not introduce sync points.<br>
Or you could look at the <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="nofollow noopener">multiprocessing best practices</a> for advice.</p>
<p>Best regards</p>
<p>Thomas</p>
Difficulty understanding entropy() in PyTorch

kabron_wade: <p>I’m new to PyTorch, and I’m having trouble interpreting entropy. Suppose, we have a probability distribution [0.1, 0.2, 0.4, 0.3]</p>
<p>First, let’s calculate entropy using numpy.</p>
<pre><code class="lang-auto">import numpy as np

p = np.array([0.1, 0.2, 0.4, 0.3])
logp = np.log2(p)
entropy1 = np.sum(-p*logp)
print(entropy1)
</code></pre>
<p>Output:</p>
<blockquote>
<p>1.846439</p>
</blockquote>
<p>Next, let’s use entropy() from torch.distributions.Categorical</p>
<pre><code class="lang-auto">import torch
from torch.distributions import Categorical

p_tensor = torch.Tensor([0.1, 0.2, 0.4, 0.3])
entropy2 = Categorical(probs = p_tensor).entropy()
print(entropy2)
</code></pre>
<p>Output:</p>
<blockquote>
<p>tensor(1.2799)</p>
</blockquote>
<p>Why is entropy1 not equal to entropy2?</p>

Mazhar_Shaikh: <p>Hi kabron_wade,<br>
The entropy is calculated using the natural logarithm. In your numpy example code, you use np.log2(). Using np.log() would give you the same result as the pytorch entropy().</p>
Backward error: an in-place operation took place and prevents backprop

John_Deterious: <p>I’m tracing back the error and it points at this:</p>
<pre><code class="lang-auto">def zlogit(re, im):
    abs_ = t.sqrt(re**2 + im**2)
    ang = t.atan2(im, re)
    mask = ~(ang &gt; 0)
    abs_[mask] = -1 * abs_[mask]
    return abs_
</code></pre>
<p>Where is the inplace operation that I did?</p>

John_Deterious: <p>I solved it! Woah! The error was misleading and was pointing to</p>
<p><code>abs_ = t.sqrt(re**2 + im**2)</code><br>
While in reality, the error is in:</p>
<pre><code>abs_[mask] = -1 * abs_[mask]
</code></pre>
<p>That was the inplace opeartion. I had to create a clone of that tensor and modify the clone like this:</p>
<pre><code>clone = abs_.clone()
clone[mask] = -1 * abs_[mask]
</code></pre>
<p>Is there a more efficient way? cause abs_ is wasted in my solution. Can the clone be a detached tensor and I go like abs_ = Dtached[mask] ? IDK.</p>

tom: <p>No, sqrt’s backward wants abs_ to calculate d sqrt(x) =  -0.5 / sqrt(x) .<br>
If you absolutely needed the memory, you use checkpointing for zlogit, but most of the time that type of memory optimization is not worth the effort.</p>
<p>Best regards</p>
<p>Thomas</p>
How to use dataloader for BRATS2015

Mosquito_ya: <p>Hello, I have <a href="https://sites.google.com/site/braintumorsegmentation/home/brats2015" rel="nofollow noopener">this dataset</a>, and I make a data loader,  it returns image, mask, and label. but when I make use it, it occurs this issue.</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test/test_dataset.py", line 16, in test_dataset
    for i, j in enumerate(dataload):
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
RuntimeError: Traceback (most recent call last):
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 68, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 68, in &lt;listcomp&gt;
    return [default_collate(samples) for samples in transposed]
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 52, in default_collate
    return default_collate([torch.from_numpy(b) for b in batch])
  File "/home/ubuntu/fzh/.conda/envs/rando/lib/python3.6/site-packages/torch/utils/data/_utils/collate.py", line 43, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: Expected object of scalar type Byte but got scalar type Short for sequence element 1 in sequence argument at position #1 'tensors'

</code></pre>
<p>and here is what I have done briefly:</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        im = []
        for i in range(self.num_input):
            direct, _ = self.root_dir[self.num_input * idx + i].split("\n")
            if i &lt; self.num_input - 1:

                image = nib.load(direct).get_data()
                image = np.expand_dims(image, axis=0)
                im.append(image)
                if i == 0 :
                    direct = os.path.split(direct)[0] + "/mask"
                    mask = nib.load(direct + "/mask.nii.gz").get_data()
            else:
                labels =nib.load(direct).get_data()
                labels = np.asarray(labels)
        iamges = np.concatenate(im, axis=0).astype(float)
        
        # iamges shape: 4 X H X W X D
        # labels shape : HXWXD
        # mask shape : HxWxD
        images = np.transpose(iamges,(0,3,1,2))
        labels = np.transpose(labels,(2,0,1))
        mask  = np.transpose(mask,(2,0,1))
        return images,labels,mask  
</code></pre>
<p>Thank you for your help</p>
How to convert .csv file to original image?

nurettinabaci: <p>Hello, I have <a href="https://www.kaggle.com/sachinpatel21/az-handwritten-alphabets-in-csv-format" rel="nofollow noopener">this dataset</a> contains the labels of english letters and its points, I want to train yolov3 with that dataset so I need to use anchor images, but before that I need to convert this file into original images. I found this <a href="https://storage.googleapis.com/kaggle-forum-message-attachments/179911/6422/gen_record.py" rel="nofollow noopener">script</a> to convert but it didn’t help me so much. Do anyone know any other way to train yolov3 with that dataset? Or help me to convert this file into original images.</p>

ptrblck: <p>There seem to be a few scripts which are loading the data and visualizing a few examples using <code>matplotlib</code>.<br>
E.g. have a look at <a href="https://www.kaggle.com/yairhadad1/cnn-for-handwritten-alphabets" rel="nofollow noopener">this script</a>, which loads the data using <code>pandas</code>.<br>
After loading, you can directly index the <code>DataFrame</code> using <code>.iloc</code> and reshape the numpy array to <code>[28, 28]</code> in order to get the appropriate shaped images.</p>
<p>If you are planning on creating a custom <code>Dataset</code>, you could load the data in the <code>__init__</code> method and get each sample in <code>__getitem__</code>.</p>
Does the ChatBot Tutorial have a bug in the masking code?

Brando_Miranda: <p>I was going over through the masking code in the chatbot tutorial and noticed that it masks with a zero on indices that are 0 but are NOT padding tokens (e.g. the first token). Is that a bug? Is the fix to use the lengths of the sequences to pad?</p>
<pre><code class="lang-auto"># Returns padded target sequence tensor, padding mask, and max target length
def outputVar(l, voc):
    '''
    padVar = padded (transposed) list of sentences with the batches
        tensor([[1391,  188,  122,   53, 5091],
        [   4,   53,   12,  154, 7708],
        [   2, 3026, 1048,  747,    4],
        [   0,    4,  115, 5747,    2],
        [   0,    2,   12, 2281,    0],
        [   0,    0, 1048,    4,    0],
        [   0,    0,    4,    2,    0],
        [   0,    0,    2,    0,    0]])
    mask = mask indicating where words occur and what isn't a word (i.e 0 for padding)
        tensor([[1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1],
        [0, 1, 1, 1, 1],
        [0, 1, 1, 1, 0],
        [0, 0, 1, 1, 0],
        [0, 0, 1, 1, 0],
        [0, 0, 1, 0, 0]], dtype=torch.uint8)
    max_target_len = length of longest target sentence
        max_target_len = 8
    '''
    # list of index representation of sentence [[124, 101, 102, 4401, 98, 382, 4, 2], ..., [67, 188, 38, 4, 2]]
    indexes_batch = [indexesFromSentence(voc, sentence) for sentence in l]
    # get length of the largest (target) sentence
    max_target_len = max([len(indexes) for indexes in indexes_batch])
    # (transposed) list of index represtations of sentences with padded zeros at the end [(124, 25, 25, 218, 67), ..., (4, 2, 0, 0, 0), (2, 0, 0, 0, 0)]
    padList = zeroPadding(indexes_batch) # padds with zeros sentences that are too long
    # returns the mask indicating which position are words locations and marks with 0 which ones are simply zeros
    st()
    mask = binaryMatrix(padList)
    #mask = torch.Tensor(padList) != PAD_token ## ALSO BUGGY?!
    mask = torch.ByteTensor(mask)
    # tensorfy (transposed) list of index represtations of sentences with padded zeros at the end. The last list is now a tensor/matrix
    padVar = torch.LongTensor(padList)
    return padVar, mask, max_target_len
</code></pre>

tom: <p>That’s not a bug, the token with index 0 <em>is</em> the padding token:</p>
<pre><code class="lang-python">PAD_token = 0  # Used for padding short sentences
</code></pre>
<p>Best regards</p>
<p>Thomas</p>

Brando_Miranda: <p>I also missed that things get transposed at some point so that confused me too (cuz I saw 2 pad tokens each in a row at the beginning of the code).</p>
Cuda::is_available() -> True with Python, false with C++

louwers: <p>When I call <code>torch::torch::cuda::is_available()</code>from Python, it returns true.<br>
However, when I call it in C++, it returns false.</p>
<p>The <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1_1cuda_1abdf5679016d4e4149f80f3fe2e80890f.html?highlight=cuda%20is_available" rel="nofollow noopener">docs</a> state that it could be a driver problem.<br>
Any idea how I can find the cause? An issue with a missling library is likely because I am running everything in a Singularity container.</p>
<pre><code class="lang-auto">[s2548259@pg-gpu ~]$ nvidia-smi
Thu Jul 18 21:28:10 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67       Driver Version: 418.67       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  On   | 00000000:03:00.0 Off |                  N/A |
|  0%   16C    P8     9W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  On   | 00000000:82:00.0 Off |                  N/A |
|  0%   22C    P8     8W / 250W |      0MiB / 11178MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>

tom: <p>Do you use the same PyTorch distribution (i.e. libtorch cmake from /usr/local/lib/python3.x/dist-packages/torch/share/cmake or somesuch)?<br>
In the end, the same libtorch should behave the same way…</p>
<p>Best regards</p>
<p>Thomas</p>

louwers: <p><a class="mention" href="/u/tom">@tom</a> I compiled the binary I was executing on a different machine, that did not have the CUDA version of PyTorch installed.</p>
<p>It’s resolved now! Thanks for your help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Extracting features from images - Triplet Model

Josiane_Rodrigues: <p>Hi,</p>
<p>I’m training a neural network model in triplet format, where I have three inputs: an anchor image and two other images, one positive and one negative in relation to the anchor image, indicating similarity.</p>
<p>I have the following difficulty: my experiments consist of extracting features for each image in the test dataset and using these features for the task of image retrieval. I’m saving the trained model, but I’m not able to load to extract the features of each image because the model is trained for a triplet format input. Some idea?</p>
Image related problem

SHASHANK_KUMAR_MISHR: <p>How to resolve the problem of combining multiple low resolution images of same object into high resolution ?</p>

ptrblck: <p>How would you like to combine them?<br>
Could you explain the use case a bit and how you would like to use PyTorch for it? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

SHASHANK_KUMAR_MISHR: <p>The problem is related to multiple image super resolution.I am not sure wether this problem can be solved by computer vision or Deep Learning.Please give some insights over it.Sharing the link below:-<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://jivp-eurasipjournals.springeropen.com/static/images/favicons/springeropen/favicon-eb9f5576a3.ico" class="site-icon" width="16" height="16">
      <a href="https://jivp-eurasipjournals.springeropen.com/articles/10.1186/s13640-016-0156-z" target="_blank" rel="nofollow noopener">EURASIP Journal on Image and Video Processing</a>
  </header>
  <article class="onebox-body">
    <img src="https://media.springernature.com/w110/springer-static/cover/journal/13640.jpg" class="thumbnail" width="" height="">

<h3><a href="https://jivp-eurasipjournals.springeropen.com/articles/10.1186/s13640-016-0156-z" target="_blank" rel="nofollow noopener">Multiple-image super-resolution on mobile devices: an image warping approach</a></h3>

<p>This paper discusses a super-resolution (SR) system implemented on a mobile device. We utilized an Android device’s camera to take successive shots and applied a classical multiple-image super-resolution (SR) technique that utilized a set of...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
How to collapse a serie of 2d convolutions

Bruno1: <p>Hi,</p>
<p>I would like to collapse a serie of 2d convolutions to a single conv operation.<br>
Basically I’d like to make the pytorch convolution associative.</p>
<p>For example if I have something like<br>
y = F.conv2d(F.conv2d(a1,k1),k0)<br>
I’d like to write it this way: y = F.conv2d(a1,K) (where K depends on k1 and k2).</p>
<p>If my calculus are not wrong, this should work:</p>
<pre><code class="lang-auto">a1 = torch.randn(1,100,256,256)
k1 = torch.randn(25,100,5,5)
k0 = torch.rand(1,25,5,5)

a0 = F.conv2d(a1,k1)
y = F.conv2d(a0,k0)

K = F.conv2d(k1.transpose(0,1),k0,padding=4).transpose(0,1)
y2 = F.conv2d(a1,K)
</code></pre>
<p>But it doesn’t work.<br>
Does anyone know what I am forgetting ? Thanks</p>

ptrblck: <p>The convolution operator in PyTorch is a cross-correlation and not a convolution in the signal processing sense.<br>
From the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d">docs</a>:</p>
<blockquote>
<p>[…] where ⋆\star⋆ is the valid 2D <a href="https://en.wikipedia.org/wiki/Cross-correlation">cross-correlation</a> operator, N is a batch size, C denotes a number of channels, H is a height of input planes in pixels, and W is width in pixels.</p>
</blockquote>
<p>While a convolution is commutative, a cross-correlation is not.<br>
Thus, you would need to flip the kernels in the spatial dimensions to get approx. the same result:</p>
<pre data-code-wrap="python"><code class="lang-python">a1 = torch.randn(1,100,256,256)
k1 = torch.randn(25,100,5,5)
k0 = torch.randn(1,25,5,5)

a0 = F.conv2d(a1,torch.flip(k1, [2, 3]))
y = F.conv2d(a0,torch.flip(k0, [2, 3]))

K = F.conv2d(k1.transpose(0, 1),torch.flip(k0, [2, 3]),padding=4).transpose(0, 1)
y2 = F.conv2d(a1,torch.flip(K, [2, 3]))

print((y - y2).abs().max())
&gt; tensor(0.0052)
</code></pre>
<p>The max abs error seems to be a bit high, but if we lower the number of channels, we can reduce this error to approx. ~1e-5, so I assume it might be due to the limited floating point precision (or I’m not seeing the bug in the code).</p>
Chatbot tutorial question

D_Liebman: <p>I’m following the chatbot tutorial at this URL.</p>
<p><a href="https://pytorch.org/tutorials/beginner/chatbot_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/chatbot_tutorial.html</a></p>
<p>I’ve written two Modules. One is a decoder and one is an encoder. They run. My problem is that when I look at my output, quite often I just see the word ‘I’ alone. In other words the chatbot thinks that it is sufficient to answer every question with the word ‘I’ alone. I donwloaded the code from the tutorial and that code does not show this behaviour.</p>
<p>I’ve tried a tensorflow project, and though it was a long time ago I believe I remember having this same problem. I am using the movie database as my corpus. I have to think someone has seen this problem before.</p>
<p>My setup currently is that the encoder processes a whole batch at a time, while the decoder goes through the batch line by line and the lines word by word. Is this my problem?</p>
<p>my code changes often and is very messy, but the url to the seq2seq model on github is here:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/radiodee1/awesome-chatbot/blob/master/model/seq_2_seq.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/radiodee1/awesome-chatbot/blob/master/model/seq_2_seq.py" target="_blank" rel="nofollow noopener">radiodee1/awesome-chatbot/blob/master/model/seq_2_seq.py</a></h4>
<pre><code class="lang-py">#!/usr/bin/python3.6
from __future__ import unicode_literals, print_function, division

import sys
#sys.path.append('..')
from io import open
import unicodedata
import string
import re
import random
import os
import torch
import torch.nn as nn
from torch.autograd import Variable
from torch import optim
import torch.nn.functional as F
import torch.nn.init as init
import time
import datetime
import math
</code></pre>

  This file has been truncated. <a href="https://github.com/radiodee1/awesome-chatbot/blob/master/model/seq_2_seq.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>any help would be appreciated.</p>

ptrblck: <p>Your code is quite long (~2800 lines of code), so that it would be helpful if you could narrow down possible issues to some functions.<br>
I’m not an NLP expert, but usually it helps me to scale down the problem, i.e. use a simple model, little dataset, and check my code for possible bugs.</p>

Brando_Miranda: <p>why don’t you write a meaningful title for your question? What issue/question? Can I know your question/issue just by reading your question title?</p>
How to set weight value to Crossentropy Loss correctly

jsong26: <p>Hi:<br>
For example, I have only two classes, 0 and 1. For class 0, I have 100 samples, for class 1 i have 900 samples. From what I understand, class 0 should have a higher weight, so would the weight [0.9,0.1] be a good approach for training?</p>

Nikronic: <p>Hi,<br>
I think <a href="https://discuss.pytorch.org/t/passing-the-weights-to-crossentropyloss-correctly/14731">this question</a> is very similar to yours. Give it a try and feel free to ask any other questions.<br>
best</p>
Torchtext integration into official pytorch docs

sidg54: <p>Hello! I was just wondering if there were any plans to add torchtext to the official documentation of pytorch rather than using the readthedocs site as it is now. It would be nice to have it integrated. I would be happy to work on this but didn’t want to start if it wasn’t part of any plan. Thank you!</p>

zhangguanheng66: <aside class="quote no-group" data-username="sidg54" data-post="1" data-topic="50869">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/eb9ed0/48.png" class="avatar"> sidg54:</div>
<blockquote>
<p>ave</p>
</blockquote>
</aside>
<p>It will be move from readthedocs to PyTorch website by the next release (early-August). It would be great if you find some missing docs and help fix it before the next release. No pressure on your side.</p>

sidg54: <p>Perfect, thanks for the information! I’ll try to contribute in some way.</p>
Is there the train model store the output in pytorch?

rofelia09: <p>Hi there,</p>
<p>In torch when we train our model for a image the model can store the output in model. Is it also possible to store the output in pytorch model after train the image?</p>
<p>In torch we can access the output after train model  as like given example …</p>
<pre><code class="lang-auto">out = model:forward(input)
</code></pre>
<p>suppose there is 10 modules in the model and I want to access the output of 5th layer in this case</p>
<p><code>layer_5 = model.modules[5].output</code></p>
<p>Like this is it possible in pytorch too. If not how will do the same in pytorch.</p>
<p>Thanks</p>

ptrblck: <p>If you would like to store the output of a certain (internal) layer, you could use forward hooks as described <a href="https://discuss.pytorch.org/t/how-can-l-load-my-best-model-as-a-feature-extractor-evaluator/17254/6">here</a>.</p>

rofelia09: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> how can I store each layer (internal or external) out put as they calculate forward pass. I am not using name of each layer I want to store according to indexing of of each layer.</p>
<p>In my model there is layer in side layer also present.</p>
<p>Example : -</p>
<pre><code class="lang-auto">Sequential(
  (0): Conv2d()
  (1): BatchNorm2d()
  (2): ReLU()
  (3): Sequential(
    (0): ConcatTable(
      (0): Sequential(
        (0): BatchNorm2d()
        (1): ReLU()
        (2): Conv2d()
        (3): BatchNorm2d()
        (4): ReLU()
        (5): Conv2d()
        (6): BatchNorm2d()
        (7): ReLU()
        (8): Conv2d()
      )
      (1): Sequential()
    )
    (1): CAddTable()
  )
</code></pre>
<p>Note : - I removed the input , output and kernel etc info.<br>
Thanks</p>

ptrblck: <p>You could iterate all modules and add the forward hooks using their name.<br>
Here is a small example:</p>
<pre><code class="lang-python">activation = {}
def get_activation(name):
    def hook(model, input, output):
        activation[name] = output.detach()
    return hook


model = nn.Sequential(
    nn.Conv2d(1, 1, 3, 1, 1),
    nn.BatchNorm2d(1),
    nn.ReLU(),
    nn.Sequential(
        nn.Conv2d(1, 1, 3, 1, 1),
        nn.BatchNorm2d(1),
        nn.ReLU()
    ),
    nn.Conv2d(1, 1, 3, 1, 1),
    nn.BatchNorm2d(1),
    nn.ReLU()
)
    
for name, module in model.named_modules():
    module.register_forward_hook(get_activation(name))

model(torch.randn(1, 1, 4, 4))

for key in activation:
    print(key, activation[key])
</code></pre>
<p>Note, that I removed <code>ConcatTable</code> and <code>CAddTable</code>, as they are classes from Torch7, which is not under active development anymore. I would therefore recommend to switch to PyTorch. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
Have a look at the <a href="https://pytorch.org/get-started/locally/" rel="nofollow noopener">website</a> for install instructions, if you haven’t already installed it.</p>
Loss.backward() crashes in RL

gioperin: <p>Hello!</p>
<p>I’m facing an issue with a reinforcement learning script, that I wrote partially inspired by this tutorial: <a href="https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html</a>. Here the author has a function for selecting the action and another one for the backprop, which makes sense because in the middle we have to perform the selected action and see the reward. However, in this tutorial in the optimization function the forward pass is called again before doing the backprop instead of saving the performed action in the buffer. Yet, I cannot do it because I am using a Bayesian Neural Network for the exploration-exploitation policy, so if I do the forward pass again, the output could be very different in terms of the action selected.<br>
It happens that if I save the predicted value in the buffer, when doing the loss.backward() it crashes unless setting retain_graph=True. In this latter case the training converges but it requires a lot of time. I don’t understand this behavior because I am not calling the function more than once, as other posts suggested. The error I get without setting retain_graph is this one:</p>
<p>RuntimeError: ntensor &gt;= 3 ASSERT FAILED at /opt/conda/conda-bld/pytorch-cpu_1556653114183/work/build/aten/src/ATen/native/cpu/IndexKernel.cpp.AVX.cpp:51, please report a bug to PyTorch.</p>
<p>If instead I call the forward pass again on the entire minibatch it works, but the training is not effective, since it is performed computing the error on couples prediction-ground_truth which may not be correlated.</p>
<p>This is what I am doing, with Transition being a namedtuple like in the tutorial:</p>
<p>transitions = memory[node].sample(minibatch)<br>
batch = Transition(*zip(*transitions))<br>
reward_batch = torch.stack(batch.reward)<br>
value_batch = torch.stack(batch.value)<br>
value_batch = value_batch.view(value_batch.shape[0],)<br>
loss = torch.nn.functional.binary_cross_entropy_with_logits(value_batch,reward_batch)<br>
optim.zero_grad()<br>
loss.backward()<br>
optim.step()</p>
How can I change pretrained resnet model to accept single channel image?

vivekstorm: <p>I’m trying to use per-trained ResNet-18 model for binary classification with modification in input channel and kernel size of 1st Conv layer. What is the best way to do this?  Also does this approach uses pretrained weight for conv1 layer?Currently I’m trying like this</p>
<pre><code class="lang-auto">model=models.resnet18(pretrained=True)
model.conv1=nn.Conv2d(1,64,kernel_size=(3,3),stride=(2,2),padding=(3,3),bias=False)
num_ftrs=model.fc.in_features
model.fc=nn.Linear(num_ftrs,2)
</code></pre>

Nishanth_Sasankan: <p>someone told me this, but I have not tried it yet</p>
<aside class="quote quote-modified" data-post="2" data-topic="48693">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/juan_sensio/48/13359_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/grayscale-images-for-resenet-and-deeplabv3/48693/2">Grayscale images for resenet and deeplabv3</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    You can always define a custom resnet and change the first layer to adapt for your input shape. For example, with resnet18: 
import torch 
import torchvision

resnet = torchvision.models.resnet18()

input = torch.randn((16,3,244,244))
output = resnet(input)
print(output.shape)

# this fails becasue resnet expects 3 channels

#input = torch.randn((16,1,244,244))
#output = resnet(input)
#print(output.shape)

import torch.nn as nn

class MyResNet(nn.Module):

    def __init__(self, in_channels=1):
…
  </blockquote>
</aside>


LeviViana: <pre><code class="lang-python">import torch
from torchvision.models import resnet18
net = resnet18(pretrained=False) # You should put 'True' here 
net.conv1 = torch.nn.Conv1d(1, 64, (7, 7), (2, 2), (3, 3), bias=False)
batch = torch.rand(4, 1, 224, 224) # in the NCHW format
net(batch).size()
</code></pre>
I want to run Libtorch (Pytorch C++ Frontend) in OpenFraneworks

bemorept: <p>HI, All.</p>
<p>I want to run Libtorch (Pytorch C++ Frontend) in OpenFraneworks.</p>
<p>LibTorch Site: <a href="https://pytorch.org/tutorials/advanced/cpp_frontend.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_frontend.html</a></p>
<p>If LibTorch is run in OpenFrameworks, It will be very powerful Deep Solution IDE(deep learning + GUI Production for OSX, Win10, &amp; Linux).</p>
<p>The setup() + update() structure of OF is very suitable for trained model’s fast inference.</p>
<p>Like this:<br>
setup() {<br>
// trained deep model loading<br>
// etc init<br>
}<br>
update() {<br>
// fast inference repeatedly of inputed.<br>
}</p>
<p><img src="https://discuss.pytorch.org/images/transparent.png" alt="dcgan" data-orig-src="upload://9vbwE5ZCv4H4XHvtzkNmsPVry1C.png" width="690" height="359"></p>
<p>But I can’t include &lt;torch/torch.h&gt; in XCode.</p>
<p>At CMake method, CMakeLists.txt is like this:</p>
<p>cmake_minimum_required(VERSION 3.0 FATAL_ERROR)<br>
project(dcgan)</p>
<p>find_package(Torch REQUIRED)</p>
<p>add_executable(dcgan dcgan.cpp)<br>
target_link_libraries(dcgan “${TORCH_LIBRARIES}”)<br>
set_property(TARGET dcgan PROPERTY CXX_STANDARD 11)</p>
<p>and install libtorch for mac: <a href="https://download.pytorch.org/libtorch/cpu/libtorch-macos-latest.zip" rel="nofollow noopener">https://download.pytorch.org/libtorch/cpu/libtorch-macos-latest.zip</a></p>
<p>How can I include libtorch header in XCode ??</p>
<p>Thanks in advance.</p>

bemorept: <p>Console version in Xcode is only Possible.</p>
<aside class="quote" data-post="1" data-topic="40233">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/r/9dc877/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/setting-up-libtorch-in-mac-xcode-ide/40233">Setting up Libtorch in Mac Xcode IDE</a> <a class="badge-category__wrapper " href="/c/c/11"><span data-category-id="11" data-drop-close="true" class="badge-category " title="Topics related to the C++ Frontend, C++ API or C++ Extensions"><span class="badge-category__name">C++</span></span></a>
  </div>
  <blockquote>
    Im new to Xcode IDE 
How to set the Cmake for libtorch in my Xcode IDE? 
So far I’ve set the 
1.OTHER_LINKER_FLAGS
2.Header search path
3.Libraries search path

However it still doesn’t work
  </blockquote>
</aside>
The purpose of requires_grad in F.l1_loss?

CgYoung: <p>why does the way of calculating l1_loss depend on target.requires_grad?</p>
<pre><code class="lang-auto">def l1_loss(input, target, size_average=None, reduce=None, reduction='mean'):
    # type: (Tensor, Tensor, Optional[bool], Optional[bool], str) -&gt; Tensor
    r"""l1_loss(input, target, size_average=None, reduce=None, reduction='mean') -&gt; Tensor

    Function that takes the mean element-wise absolute value difference.

    See :class:`~torch.nn.L1Loss` for details.
    """
    if not (target.size() == input.size()):
        warnings.warn("Using a target size ({}) that is different to the input size ({}). "
                      "This will likely lead to incorrect results due to broadcasting. "
                      "Please ensure they have the same size.".format(target.size(), input.size()),
                      stacklevel=2)
    if size_average is not None or reduce is not None:
        reduction = _Reduction.legacy_get_string(size_average, reduce)
    if target.requires_grad:
        ret = torch.abs(input - target)
        if reduction != 'none':
            ret = torch.mean(ret) if reduction == 'mean' else torch.sum(ret)
    else:
        expanded_input, expanded_target = torch.broadcast_tensors(input, target)
        ret = torch._C._nn.l1_loss(expanded_input, expanded_target, _Reduction.get_enum(reduction))
    return ret
</code></pre>
I made a t-SNE plot code using Openframeworks & XCode

bemorept: <p>HI, All.</p>
<p>I made a t-SNE plot code using Openframeworks &amp; XCode.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/bemoregt/twoClassTSNE">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/bemoregt/twoClassTSNE" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/28c21c082febd0c8ba00b5f32e62f06fa1cdad70a56a22bf9dfa01cfe2843f99/bemoregt/twoClassTSNE" class="thumbnail" width="" height=""></div>

<h3><a href="https://github.com/bemoregt/twoClassTSNE" target="_blank" rel="noopener nofollow ugc">GitHub - bemoregt/twoClassTSNE: Pre-Trained Imagenet based two class T-SNE...</a></h3>

  <p>Pre-Trained Imagenet based two class T-SNE plot using Openfreameworks - GitHub - bemoregt/twoClassTSNE: Pre-Trained Imagenet based two class T-SNE plot using Openfreameworks</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This code use pre-trained imagenet model for features mapping.<br>
So, It is a kind of semi-unsupervised clustering.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/4/4d1762f8dfad2ed3bfa59878e97249a9ceacacb5.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d1762f8dfad2ed3bfa59878e97249a9ceacacb5" title="ScrShot%2012-1"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4d1762f8dfad2ed3bfa59878e97249a9ceacacb5_2_629x500.png" alt="ScrShot%2012-1" data-base62-sha1="aZYRZnjOBGi3Is75ulwWqLXaryd" width="629" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/4/4d1762f8dfad2ed3bfa59878e97249a9ceacacb5_2_629x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/4/4d1762f8dfad2ed3bfa59878e97249a9ceacacb5_2_943x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/4/4d1762f8dfad2ed3bfa59878e97249a9ceacacb5.png 2x" data-dominant-color="565352"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ScrShot%2012-1</span><span class="informations">1136×902 262 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>But, It uses just VGG16.sqlite model format.<br>
How can I Convert my pytorch model to *.sqlite format ??</p>
<p>I wonder.</p>
<p>Thanks.<br>
Best,<br>
<span class="mention">@bemoregt</span>.</p>