akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Loss function of embedded vectors

username: helloWorld
cooked: <p>I have an app that uses <a href="https://github.com/timesler/facenet-pytorch/blob/master/models/utils/training.py" rel="nofollow noopener">facenet-pytorch</a> to generate embedded vectors for images of faces. The model I’m using is InceptionResnetV1 trained on vggface2. The output layer of this model is a 512d tensor.</p>
<p>I’m trying to write a TFGSM attack that will allow me to send an Image to the resnet model and get back an embeddeing vector that is simillar to embedded vector of someone specific.</p>
<p>For example, I took 10 images of Eylon Musk , passed them to the model and got 10 embedded vectors. I created one average embedded vector from all of those vectors and this new vector is an embedded vector that represents Eylon Musk. I have an image of Barak Obama` face. I want to run an TFGSM attack that will change some pixels in the image , so that when I’ll forward this image in the resnet model I will get a vector that is simillar to the vector of Eylon Mask(L2 distance will be less than some threshold…).<br>
I got 2 questions :</p>
<p>I’m trying to understand what  <strong>LOSS function</strong> should I use. I checked CosineSimilarity and CosineEmbeddingLoss . I’m not sure why I need the y parameter in COsineMbeddingLoss since I got 2 vectors… And what is the main difference between them ?</p>
<p>My attack code :</p>
<pre><code class="lang-auto">def TFGSM(image:torch.Tensor, model, target_vector,epsilon):
    loss = nn.CosineSimilarity()
    #loss = nn.CosineEmbeddingLoss()
    loss = loss(model(image), target_vector)
    #loss = loss(model(image), target_vector,torch.Tensor([[1]*512]))
    model.zero_grad()
    loss.backward()
    data_grad = image.grad.data
    sign_data_grad = data_grad.sign()
    image_with_noise = image + epsilon*sign_data_grad
    image_with_noise = torch.clamp(image_with_noise, 0, 1)
    return image_with_noise
</code></pre>
<p>It doesnt matter If I’m using the CosineEmbeddinngLoss or the CosineSimilarity I’m getting the following exception in both case :<br>
<code>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</code><br>
Tried adding retain_graph=True but it didnt help.</p>

username: helloWorld
cooked: <p>Any idea maybe why I’m getting the RuntimeError ?</p>

username: ptrblck
cooked: <p>Which line of code is raising this error?</p>
<p>This should be unrelated to the current error message, but note that you override the <code>loss</code> criterion with the <code>loss</code> tensor, which might yield other issues, if you are trying to call the same criterion again.</p>
Expanding pretrained embedding

username: barthelemymp
cooked: <p>Hello,</p>
<p>On my current project I’m using the google word2vec embedding googlenews-vectors-negative300.bin<br>
However I was surprised that a lot of word in my text are nor referenced in the embedding(like xenophobia, submissive etc).</p>
<p>Firstly, I wanted to know how I can extand a  <code>nn.Embedding</code> with new words. I guess I should then activate backpropagation on this part of the embedding for it to be learned.</p>
<p>Secondly, I don’t know why but I need to pass by gensim to load the embedding, Indeed</p>
<pre><code class="lang-auto">text_field = data.Field(sequential=True, tokenize=_tokenize_str)
dataset = TabularDataset(
    path='mydata.csv',
    format='csv',
    fields=[('id',None),('content',text_field )],
    skip_header=False)
text_field.build_vocab(dataset)
vectors = vocab.Vectors('/data/GoogleNews-vectors-negative300.bin.gz')
text_field.vocab.set_vectors(vectors.stoi, vectors.vectors, vectors.dim)
embedding = nn.Embedding.from_pretrained(torch.FloatTensor(text_field.vocab.vectors))
</code></pre>
<p>does not work, instead I need to do first:</p>
<pre><code class="lang-auto">model = gensim.models.KeyedVectors.load_word2vec_format('data/GoogleNews-vectors-negative300.bin.gz', binary=True)
model.wv.save_word2vec_format('data/myGoogleEmbedding.bin')
vectors = vocab.Vectors('/content/drive/My Drive/ActNews/data/myGoogleEmbedding.bin') 
text_field.vocab.set_vectors(vectors.stoi, vectors.vectors, vectors.dim)
embedding = nn.Embedding.from_pretrained(torch.FloatTensor(text_field.vocab.vectors))
</code></pre>
<p>Best regards,</p>
<p>Barthélémy</p>

username: ptrblck
cooked: <p>You could try to concatenate the pretrained weight matrix with a newly initialized tensor to create the new weight matrix with the extended vocabulary.<br>
To keep the pretrained embedding matrix constant, you could register a hook to zero out the gradients of this part of the <code>weight</code>.<br>
Here is a small code snippet to demonstrate this approach:</p>
<pre><code class="lang-python">vocab_size = 2
embedding_dim = 10
emb = nn.Embedding(vocab_size, embedding_dim)

# Add vocab
emb.weight = nn.Parameter(
    torch.cat((emb.weight, torch.randn(2, embedding_dim))))

# Register hook to zero out gradients of pretrained embedding weights
mask = torch.zeros_like(emb.weight)
mask[2:] = 1.
emb.weight.register_hook(lambda grad: grad*mask)

# Training
x = torch.randint(0, 4, (10,))
out = emb(x)
out.mean().backward()

# Should pring zeros in first half
print(emb.weight.grad)
</code></pre>
<p>Let me know, if this would work for you.</p>
Speeding up a sum of products (that is not a matmul)

username: bluehood
cooked: <p>Hi,<br>
my application spends most of its time doing a particular tensor operation:</p>
<pre><code class="lang-python">&gt;&gt;&gt; a # shape (..., N, M)
&gt;&gt;&gt; b # shape (N, M)
&gt;&gt;&gt; (a*b).sum(dim=-1)
</code></pre>
<p>With <code>einsum</code>, the operation can be written as</p>
<pre><code class="lang-python">&gt;&gt;&gt; torch.einsum('...ij,ij-&gt;...i', (a,b))
</code></pre>
<p>Einsum is actually faster (maybe because it doesn’t need to allocate the temporary <code>a*b</code>?) in all of my tests, both on CPU and GPU…</p>
<p>Since this particular computation is <em>the</em> bottleneck in my application, I thought I’d check whether someone here knows how to speed it up even more – I’m not sure whether jitting would help (never used it)?</p>
<p>Thanks in advance,<br>
Enrico</p>
<p>EDIT: permutated versions of the tensors are free: I can produce these tensors with whatever shapes make computation faster.</p>

username: bluehood
cooked: <p>Bump – I promise to not bump again</p>

username: Vector
cooked: <p>Any luck speeding it up? I have a similar us case…</p>
How to use visual studio code for pytorch

username: spnova12
cooked: <p>I want use visual studio code for pytorch.</p>
<p>it is working…?</p>
<p>but torch is underlined in red.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/d/d1f81a123fb99860b52674c8a84a0884d3d337b3.png" alt="image" data-base62-sha1="tXtlw7GAmqCszZA8sJIOkdDXbzl" width="497" height="224"></p>
<p>How can i delete that red line…<img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: ashafaei
cooked: <p>An option is to remove <em>torch</em> from Pylint. Based on a <a href="https://stackoverflow.com/questions/20553551/how-do-i-get-pylint-to-recognize-numpy-members" rel="nofollow noopener">similar solution for numpy</a>, you can add the following config to your settings.</p>
<pre><code class="lang-auto">{
    // whitelist torch to remove lint errors
    "python.linting.pylintArgs": [
        "--extension-pkg-whitelist=torch"
    ]
}
</code></pre>
<p>A more proper approach might be to fix the Pylint issue with PyTorch, but I don’t know what’s exactly causing this problem (maybe torch functions are not statically resolvable).</p>

username: spnova12
cooked: <p>Thank you very much!!</p>

username: gt_tugsuu
cooked: <p>Or you can add <code>torch</code> to Pylint.</p>
<pre><code class="lang-auto">{
    "python.linting.pylintArgs": [
        // "--generated-members=torch.* "
        ],
}
</code></pre>

username: navidre
cooked: <p>You may want to give flake8 a try as well. Uninstall Pylint and install flake8 using:</p>
<pre><code class="lang-auto">pip install flake8 flake8-mypy flake8-bugbear flake8-comprehensions flake8-executable flake8-pyi mccabe pycodestyle pyflakes
</code></pre>
<p>or just:</p>
<pre><code class="lang-auto">pip install flake8
</code></pre>
<p>Reference: <a href="https://github.com/pytorch/pytorch/wiki/Lint-as-you-type" rel="nofollow noopener">https://github.com/pytorch/pytorch/wiki/Lint-as-you-type</a></p>
Save a list of tensos into a file

username: navid_mahmoudian
cooked: <p>Suppose that I have a list of tensors (the tensors have different sizes), and I want to save this list into a single file. Is there any way to do this? The tensors are not model parameters and they are just some data.</p>

username: albanD
cooked: <p>Hi,</p>
<p>You can just <code>torch.save()</code> the whole list. It will create a single file with the list.</p>

username: navid_mahmoudian
cooked: <p>Thank you very much for your quick response</p>
Weighted sampling in torchtext

username: iqopi
cooked: <p>I am currently using torchtext to load a NLP dataset, and i want so oversample some classes (repeat samples of specific classes more often while training).<br>
My code currently looks something like this:</p>
<pre><code class="lang-auto">tokenize = lambda x: x.split()

TEXT = Field(sequential=True, tokenize=tokenize, lower=True, unk_token = None)
LABEL = Field(sequential=False, use_vocab=True, unk_token = None)

fields = [('id', None), ('text', TEXT), ('label',LABEL)]

training, validation = TabularDataset.splits(
                        path = "./",
                        train = "training.csv", validation = "validation.csv",
                        format = "csv",
                        skip_header = True,
                        fields = fields)

TEXT.build_vocab(training, max_size = None)
LABEL.build_vocab(training)

train_iter, val_iter, _ = data.BucketIterator.splits((
                    training, validation, _),
                    batch_size = batchsize,
                    device = device,
                    sort_key = lambda x: len(x.text))
</code></pre>
<p>I have read here <a href="https://discuss.pytorch.org/t/balanced-sampling-between-classes-with-torchvision-dataloader/2703/3" class="inline-onebox">Balanced Sampling between classes with torchvision DataLoader</a> that you can pass on a <code>sampler</code> argument to PyTorchs <code>data.DataLoader</code> function.</p>
<p>I there something similar that would work for my use case?</p>

username: Shanto_Islam
cooked: <p>I am also stuck on the same place. Did you got a fix?</p>
Discussion on combining sub-state into one state for training rl model

username: nlpdl
cooked: <p>Hello,</p>
<p>Would like to ask the community, if anyone has suggestion on representing a single state from multiple sub-states for rl training.</p>
<p>For example, I have a substate that is represented by a 2d tensor (width by height). But then i have another substate represented as a vector of <em>n</em> inputs.</p>
<p>With these two sub-states, I would like to combine them to represent 1 input state to my neural architecture.</p>
<p>I’m not sure if it would be valid to perhaps convert the 2d tensor substate into a 1d tensor and append the other vector input to it?</p>
<p>Appreciate any suggestions or advice, tku!</p>

username: kengz
cooked: <p>Take an image and a vector as example, you cannot combine both tensors directly; instead you may treat them as different modes, and have a multi-head network talking them as inputs:</p>
<ul>
<li>im_head: have a conv module with the image as input</li>
<li>vec_head: have a plain mlp module with the vector as input</li>
</ul>
<p>Now you can explore a few ways of combining them:</p>
<ol>
<li>flatten the im_head output, then concat it with the vec_head output, or</li>
<li>flatten the im_head output, add a linear so that it comes out the same shape as vec_head, then do element-wise addition</li>
<li>more generally, taking from 2, you can do FiLM: <a href="https://distill.pub/2018/feature-wise-transformations/" rel="nofollow noopener">https://distill.pub/2018/feature-wise-transformations/</a>
</li>
</ol>
<p>Once combined, you can further pass through more MLPs then to whatever output architecture you need.</p>
Tensor creation slow on cpu (from replay buffer)

username: floepfl
cooked: <p>Hello, I’m implementing Deep Q-learning and my code is slow due to the creation of Tensors from the replay buffer. Here’s how it goes:</p>
<p>I maintain a  <code>deque</code>  with a size of 10’000 and sample a batch from it everytime I want to do a backward pass. The following line is really slow:</p>
<p><code>curr_graphs = torch.Tensor(list(state(*zip(*xp_samples.curr_state)).graph))</code></p>
<p>That I decomposed to see what is really taking time</p>
<p><code>zipped = zip(*xp_samples.curr_state)</code><br>
<code>new_s = state(*zipped)</code><br>
<code>listt = list(new_s.graph)</code><br>
<code>curr_graphs = torch.Tensor(listt)</code></p>
<p>To notice that the last line, i.e the tensor creation, is what is taking all the computation. What is happening is that xp_samples and curr_state are named tuples. In this snippet I unpack and then zip and then unpack again to group the data by name from curr_state.</p>
<p>In my opinion it has to assemble data from memory using a lot of pointers to create the Tensor and thus is losing time moving things around. What would be the fastest way to create a tensor from sampled data from a buffer that I maintain? Should I allocate the size of the content of the deque so that it is continous in memory? I feel like it won’t speed up the process.</p>
<p>Here are the details of the <code>deque</code> if that’s relevant:</p>
<p><code>class ReplayBuffer:</code><br>
<code>    def __init__(self, maxlen):</code><br>
<code>        self.buffer = deque(maxlen=maxlen)</code><br>
<code>  </code><br>
<code>    def add(self, new_xp):</code><br>
<code>        self.buffer.append(new_xp)</code><br>
<code>      </code><br>
<code>    def sample(self, batch_size):</code><br>
<code>        xps = random.choices(self.buffer, k=batch_size)</code><br>
<code>        return xp(*zip(*xps))</code></p>

username: kengz
cooked: <p>Sample indices directly, use numpy arrays, or store as tensors directly. Please see this similar question and its answer: <a href="https://discuss.pytorch.org/t/how-to-make-the-replay-buffer-more-efficient/80986/2" class="inline-onebox">How to make the replay buffer more efficient？</a></p>
TUTORIAL DQN example: NoSuchDisplayException in Colab

username: droogg
cooked: <p>I try to run “REINFORCEMENT LEARNING (DQN) TUTORIAL” in Colab and get a NoSuchDisplayException: Cannot connect to “None”.</p>
<blockquote>
<h2>env.reset()<br>
plt.figure()<br>
plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),<br>
interpolation=‘none’)<br>
plt.title(‘Example extracted screen’)<br>
plt.show()</h2>
<p>NoSuchDisplayException                    Traceback (most recent call last)<br>
 in ()<br>
39 env.reset()<br>
40 plt.figure()<br>
—&gt; 41 plt.imshow(get_screen().cpu().squeeze(0).permute(1, 2, 0).numpy(),<br>
42            interpolation=‘none’)<br>
43 plt.title(‘Example extracted screen’)</p>
<p>10 frames<br>
/usr/local/lib/python3.6/dist-packages/pyglet/canvas/xlib.py in <strong>init</strong>(self, name, x_screen)<br>
84         self._display = xlib.XOpenDisplay(name)<br>
85         if not self._display:<br>
—&gt; 86             raise NoSuchDisplayException(‘Cannot connect to “%s”’ % name)<br>
87<br>
88         screen_count = xlib.XScreenCount(self._display)</p>
<p>NoSuchDisplayException: Cannot connect to “None”</p>

</blockquote>
<p>I thought that the tutorials is adapted to the Colab, because there is even a link to launch them. What is an easy way to avoid this error? Or is it necessary to deal with pyvirtualdisplay?</p>

username: abyaadrafid
cooked: <p>Still facing the issue. Did you find any workaround?</p>

username: kengz
cooked: <p>Add a line above your notebook to plot inline:<br>
<code>%matplotlib inline</code></p>
<p>source: <a href="https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/04.00-Introduction-To-Matplotlib.ipynb#scrollTo=0YPlWWlcucAx" rel="nofollow noopener">https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/04.00-Introduction-To-Matplotlib.ipynb#scrollTo=0YPlWWlcucAx</a></p>
Model doesn't give high probability though it's accuracy is 85%

username: Mahmoud_Housam
cooked: <p>Hello dears, I’ve built my first deep learning model using CNN. My issue is with too low probabilities I get from the model though the model accuracy is 85% roughly.<br>
I’ve tried many other photos but nothing helped out. May you help me please where the issue is?<br>
Here is the project notebook<br>
<a href="https://github.com/MahmoudHousam/Finding-Donors-for-ChartiyML/blob/master/Image%20Classifier%20Project.ipynb" rel="nofollow noopener">Deep Learning Model Classifier</a></p>

username: Mahmoud_Housam
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> May you help me please?</p>

username: Nikronic
cooked: <p>Hi,</p>
<p>Can you report probabilities for a sample test/train image?<br>
And also, Could you please explain why are you using log(softmax)? For comparison purposes or in your case, softmax is a more interpretable choice. Softmax maps input to probability space in range of [0, 1] so applying log on top of it, will lead to a all negative numbers but sum of them won’t be 1.</p>
<p>Bests</p>

username: Mahmoud_Housam
cooked: <p>using softmax instead of log(softmax) giving constant accuracy</p>

username: Nikronic
cooked: <p>That is strange.</p>
<p>But about probabilities. Could you please explain how probabilities are not high enough? I mean, what is your comparison metric or let’s say for a image x, what do you expect and what are probs currently you obtaining?</p>
Install from source for old GPU support. Too new gcc version - no admin rights

username: Peter1998
cooked: <p>Hey, I want to install from source on a Linux machine with a Quadro K4000 GPU and Cuda 9.2 in a conda environment. I followed <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">these instructions</a>, however, in the last step <em>python setup.py install</em> I get many error messages of this type:</p>
<blockquote>
<p>error: <span class="hashtag-raw">#error</span> – unsupported GNU version! gcc versions later than 7 are not supported!<br>
<span class="hashtag-raw">#error</span> – unsupported GNU version! gcc versions later than 7 are not supported!</p>
</blockquote>
<p><em>gcc --version</em> shows that I have version 8.3.0. I found similar problems here, but the suggested solutions require admin rights, which I don’t have. I was hoping to install an older gcc version via conda (from here on I’m on really thin ice and don’t really know what I am doing). I am not sure which one to use, so I randomly chose <em>conda install -c rgrout gcc_linux-64</em>. At least the results of <em>$CC --version</em> is promising: <em>x86_64-conda_cos6-linux-gnu-cc (crosstool-NG fa8859cb) 6.4.0</em></p>
<p>To my understanding, I would need to tell the cmake to use ‘my’ gcc via $CC instead of the system one. Is this a promising approach? And how would I do this step? Thanks</p>
<p>EDIT: I tried <em>conda install gcc_linux-64=5.4.0</em> now and then followed the <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">instructions</a> without explicitly specifying to use the conda gcc. I get this:</p>
<pre><code class="lang-auto">The C++ compiler

    "/my_home_path/anaconda3/envs/pt/bin/x86_64-conda_cos6-linux-gnu-cc"

  is not able to compile a simple test program.

  It fails with the following output:

    Change Dir: /my_home_path/pytorch/build/CMakeFiles/CMakeTmp
    
    Run Build Command(s):/my_home_path/anaconda3/envs/pt/bin/ninja cmTC_4e2b1 
    [1/2] Building CXX object CMakeFiles/cmTC_4e2b1.dir/testCXXCompiler.cxx.o
    FAILED: CMakeFiles/cmTC_4e2b1.dir/testCXXCompiler.cxx.o 
    /my_home_path/anaconda3/envs/pt/bin/x86_64-conda_cos6-linux-gnu-cc    -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -O2 -pipe -o CMakeFiles/cmTC_4e2b1.dir/testCXXCompiler.cxx.o -c testCXXCompiler.cxx
    x86_64-conda_cos6-linux-gnu-cc: error trying to exec 'cc1plus': execvp: No such file or directory
    ninja: build stopped: subcommand failed.
</code></pre>
When set requires_grad() False, tensor grad is None

username: JIE_LIU
cooked: <p>Hi, I have a problem about autograd. The code is below:</p>
<pre><code class="lang-python">from torch import nn
input = torch.randn(8, 3, 50, 100)

net = nn.Sequential(nn.Conv2d(3, 16, 3, 1),nn.Conv2d(16, 32, 3, 1))

net.named_parameters().__next__()[1].requires_grad = False

for param in net.named_parameters():
    print(param[0], param[1].requires_grad)

output = net(input)

net.named_parameters().__next__()[1].requires_grad = True

torch.mean(output).backward()
for param in net.named_parameters():
    param[0], param[1].grad
</code></pre>
<p>then part of output is listed</p>
<pre><code class="lang-auto">0.weight False 
0.bias True 
1.weight True 
1.bias True
('0.weight', None)
</code></pre>
<p>I set requires_grad() False for 0.weight, then I set requires_grad() True before backward. But I still get None for 0.weight grad. I’m wondering what step in forward cause this problem.</p>

username: ptrblck
cooked: <p>Since you are resetting the <code>requires_grad</code> attribute to <code>True</code> after the forward pass, so after the computation graph was already created, you most likely won’t get the gradient calculation for this particular parameter anymore.<br>
Try to reset it before the forward pass and check the <code>grad</code> again.</p>

username: JIE_LIU
cooked: <p>Thanks. But I’m still curious about what in computation graph cause this problem. If I set the <code>requires_grad</code> to <code>False</code> for this tensor, will it not store the intermediate calculation result?</p>

username: ptrblck
cooked: <p>Yes, if some parameters and thus operations don’t need gradients, the intermediate tensors won’t be stored. The same would happen, if you wrap the code in a <code>with torch.no_grad()</code> block to save memory during the validation pass.</p>
Lower limit in learning rate in PyTorch torch.optim

username: mohit117
cooked: <p>I was going through the <a href="https://stackoverflow.com/a/55027576/13049379" rel="nofollow noopener">following information</a> on reducing learning rates in PyTorch to really low value like 1e-9.</p>
<p>And I am amazed why doing <code> loss = loss/100</code> is equivalent to reducing learning rate by 100? The full snippet is below.</p>
<pre><code class="lang-auto">outputs = model(batch)
loss = criterion(outputs, targets)

# Equivalent to lowering the learning rate by a factor of 100
loss = loss / 100

self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
</code></pre>
<p>Can someone please help me on this.</p>
<p>Thanks a lot</p>

username: ptrblck
cooked: <p>If you scale the loss, you’ll also scale the gradients.<br>
In a simple use case, this can be used instead of changing the learning rate, as seen here:</p>
<pre><code class="lang-python"># Setup
torch.manual_seed(2809)

lin = nn.Linear(2, 2, bias=False)
x = torch.randn(1, 2)

# Standard approach
out = lin(x)
loss = out.sum()
print(loss)
loss.backward()
print(lin.weight.grad)
&gt; tensor(-0.8130, grad_fn=&lt;SumBackward0&gt;)
tensor([[-1.1281,  0.8386],
        [-1.1281,  0.8386]])

# loss scaling by x10
lin.zero_grad()
out = lin(x)
loss = out.sum() * 10
print(loss)
loss.backward()
print(lin.weight.grad)
&gt; tensor(-8.1301, grad_fn=&lt;MulBackward0&gt;)
tensor([[-11.2812,   8.3855],
        [-11.2812,   8.3855]])

# loss scaling by x0.1
lin.zero_grad()
out = lin(x)
loss = out.sum() * 0.1
print(loss)
loss.backward()
print(lin.weight.grad)
&gt; tensor(-0.0813, grad_fn=&lt;MulBackward0&gt;)
tensor([[-0.1128,  0.0839],
        [-0.1128,  0.0839]])
</code></pre>
<p>However, I would be careful with more advanced optimizers, which are e.g. tracking the running average of the gradients or are adjusting the learning rate internally somehow.</p>
Convert ndarray of tensors into tuple of tensors with gradients

username: ku294714
cooked: <p>I have a nd array of tensors with gradeints:<br>
lis = [0,1,2]<br>
ae = torch.tensor(lis, dtype = torch.float, requires_grad=True)<br>
aqw =np.array([ae[0], ae[1],ae[2]])</p>
<p>I need to convert into a tensor with gradients intact. I tried torch.from_numpy but it losses the grads and I want to have a use case like this, I do not want to perform my computations in torch but in numpy</p>

username: ptrblck
cooked: <p>Numpy does not provide an autograd engine, so you won’t be able to automatically differentiate your operations in numpy.<br>
If you want to use numpy methods in the forward pass, you could implement the backward manually by writing a custom <code>torch.autograd.Function</code> as described <a href="https://pytorch.org/tutorials/beginner/examples_autograd/two_layer_net_custom_function.html">here</a>.</p>
Calc grad of tensor that isnt leaf tensor

username: helloWorld
cooked: <p>Hey,<br>
I’m using the following code in order to get the gradients of the tensor calculated. I want to change the tensor values in aspect to the loss:</p>
<pre><code class="lang-auto">&gt; from facenet_pytorch import InceptionResnetV1
&gt; import torch.nn as nn
&gt; resnet = InceptionResnetV1(pretrained='vggface2').eval()
&gt; tensor1 = torch.tensor((), dtype=torch.float32)
&gt; tensor2 = torch.tensor((), dtype=torch.float32)
&gt; tensor1 = torch.stack([(tensor1.new_ones(3,160,160))])
&gt; tensor2 = torch.stack([(tensor2.new_ones(3,160,160))])
&gt; tensor1.require_grad=True
&gt; ev1 = resnet(tensor1)
&gt; ev2 = resnet(tensor2)
&gt; print(ev1.shape) # torch.Size([1, 512])
&gt; cos = nn.CosineSimilarity()
&gt; loss = cos(ev1, ev2)
&gt; loss.backward(retain_graph=True)
&gt; data_grad = ev1.grad.data
&gt; print(data_grad)
</code></pre>
<p>It seems that loss.backwards doesnt calc anything because in the next line</p>
<pre><code class="lang-auto">(ev1.grad.data) grad is None : 
     15 loss.backward(retain_graph=True)
---&gt; 16 data_grad = ev1.grad.data

AttributeError: 'NoneType' object has no attribute 'data'
</code></pre>
<p>I also got a warning that I try to access .grad attribute of a tensor that isnt a leaf tensor. So how can I calc the grad of the orig tensor ?</p>

username: ptrblck
cooked: <p>The error message should also mention, that you can use <code>retain_grad()</code> on the non-leaf variable to get the gradient.</p>
<pre><code class="lang-python">ev1 = resnet(tensor1)
evt.retain_grad()
...
</code></pre>
<p>should therefore work.</p>
<p>Unrelated to this issue, but you should use <code>.requires_grad</code> (with an s) and you should not use the <code>.data</code> attribute as it might yield unwanted side effects.</p>

username: helloWorld
cooked: <p>Thanks ! I think that in my case, I wanted to retain_grad of the orig tensor(tensor1) and not of the embedded vector (ev1). It should work the same right ?</p>

username: ptrblck
cooked: <p>It depends, which tensor should get the gradients.<br>
If you want to get gradients for <code>tensor1</code>, you should use</p>
<pre><code class="lang-python">tensor1.requires_grad = True
</code></pre>

username: helloWorld
cooked: <p>Thank you very much !</p>
How to generate random number on GPU

username: guyue_hu
cooked: <p>Hi all,<br>
how to generate  random number on GPU, because I find generate a big rand tensor on CPU and then transform it into cuda tensor (<code>a= torch.randn(1000,512,20,20); a.cuda()</code>)<br>
is really CPU comsuming. Is any to generate it on GPU not CPU?Thank you advance!</p>

username: richard
cooked: <p>The <a href="http://pytorch.org/docs/master/torch.html?highlight=rand#torch.randn" rel="nofollow noopener">randn docs</a> suggest that <code>randn</code> can take an <code>out</code> argument. You can do something like</p>
<pre><code class="lang-auto">shape = torch.Size((300, 300))
x = torch.cuda.FloatTensor(shape)
torch.randn(shape, out=x)
</code></pre>

username: guyue_hu
cooked: <p>Yes, that what I am looking for, thank you a lot!</p>

username: gokul_uf
cooked: <p>Doesn’t this still result in a data copy to the GPU? What if I want to generate directly on the GPU?</p>

username: nofreewill
cooked: <p>Did this speed up the generation process?</p>
How to feed softmax output to embedding as rnn input

username: Mohammad_Javad
cooked: <p>I’m implementing this network:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a08f485a096099f5a0398185e2ba0ebb9febe39.png" data-download-href="https://discuss.pytorch.org/uploads/default/5a08f485a096099f5a0398185e2ba0ebb9febe39" title="1_SWWZVOXS6vpL9CEIpPntQw"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a08f485a096099f5a0398185e2ba0ebb9febe39_2_690x233.png" alt="1_SWWZVOXS6vpL9CEIpPntQw" data-base62-sha1="cQu9Q0sg0RkBaaSIO8qCj8DG8Ot" width="690" height="233" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a08f485a096099f5a0398185e2ba0ebb9febe39_2_690x233.png, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a08f485a096099f5a0398185e2ba0ebb9febe39.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a08f485a096099f5a0398185e2ba0ebb9febe39.png 2x" data-dominant-color="F2F2EE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1_SWWZVOXS6vpL9CEIpPntQw</span><span class="informations">786×266 21.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
how can I feed softmax output to the embedding layer?</p>

username: crcrpar
cooked: <p>How about applying <code>argmax</code> to the output of softmax as follows?</p>
<pre><code class="lang-python"># -*- coding: utf-8 -*-
"""Untitled3.ipynb

Automatically generated by Colaboratory.

Original file is located at
    https://colab.research.google.com/gist/crcrpar/adecbb7a838e62a081a05a9cd53be048/untitled3.ipynb
"""

import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):

    def __init__(self, num_classes=10, dim_embed=128):
        super().__init__()
        self.softmax = nn.Softmax(1)
        self.embed = nn.Embedding(num_embeddings=num_classes, embedding_dim=dim_embed)

    def forward(self, x):
        prob = self.softmax(x)
        indices = torch.argmax(prob, dim=1)
        return self.embed(indices)

model = Model()
# N: 32, C: 10
sample_input = torch.randn(32, 10)
# This `no_grad` shouldn't be in your training code :slightly_smile:
with torch.no_grad():
    embeddings = model(sample_input)
print(embeddings.shape)
</code></pre>
How to apply negative sign to some samples in mini-batch?

username: oasjd7
cooked: <p>What should I do to get some samples to be trained as usual (classified well)<br>
and some samples to trained the opposite way(not to be classified well)  in a mini-batch?</p>
<p>For example, In mini-batch, (batch_size = 3)<br>
[1st img,  2nd img,  3rd img]</p>
<p>For 1st &amp;3rd img =&gt; classified well as usual.<br>
But for 2nd =&gt; not to be classified well.</p>
<p>1st img -&gt; get loss -&gt; loss<br>
2nd img -&gt; get loss -&gt; loss * (-1)<br>
3rd img -&gt; get loss -&gt; loss<br>
total_losses = sum of three-losses</p>
<p>Is this way works?</p>
How to obtain cordinates at which abrupt chnage in pixel occurs?

username: Ratan
cooked: <p>I have an image in which there is a a white box. moving from top to bottom(height) I want to get the coordinates of first horizontal line and bottom horizontal line of the box using sudden change in pixel values<br>
How do achieve this?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86884d0e628c4b499fc116b117945e98139fe2a5.png" alt="Screenshot 2020-05-28 at 11.34.07 AM" data-base62-sha1="jc80hDk4Kw3XRAKNFqYW07yFlCB" width="281" height="238"></p>

username: ptrblck
cooked: <p>I’m not sure, if you would like to use a neural network for it, as you are posting in the PyTorch forum, but you could use standard image processing to get these lines.<br>
To do so you could use a horizontal edge detection, e.g. using a <a href="https://en.wikipedia.org/wiki/Sobel_operator">sobel operator</a>, and afterwards filter the image for these two lines (e.g. using a <a href="https://en.wikipedia.org/wiki/Matched_filter">matched filter</a>).</p>
<p>This could also be done using a neural network (which might be a bit overkill, but seems to be a fun task), so let me know, which approach you would like to take.</p>

username: Ratan
cooked: <p>Yes i did few image processing hack to get the results.<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a> I would like to know how it can be done with using a neural network!</p>

username: ptrblck
cooked: <p>You could use your image processing approach to create the target values (e.g. as pixel locations of the start and end coordinates of the wanted lines or alternatively as a mask image) and start building a CNN to predict these targets.</p>
<p>I don’t know which target format and architecture would work the best, so you would have a lot to experiment with. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
AttributeError: module 'torch.utils.data' has no attribute 'IterableDataset'

username: jalajthanaki
cooked: <p>Hi Team,<br>
I’m using fairseq and PyTorch 1.5 version and I’m facing the following issue while running interactive.py script.</p>
<p><code>AttributeError: module 'torch.utils.data' has no attribute 'IterableDataset'</code></p>
<p>On <a href="https://github.com/pytorch/fairseq/issues/1437" rel="nofollow noopener">this Fairseq GitHub issue</a> Fairseq team suggested to raise an question here so doing the same.</p>
<p>Please let me know if I can resolve this issue. Is there anything I’m missing? Any help or suggestion is appreciable.</p>
<p>Thanks</p>

username: albanD
cooked: <p>Hi,</p>
<p>What does <code>print(torch.__version__)</code> returns just before your try to import fairseq?<br>
You might have an older version of pytorch somewhere on your system.</p>

username: jalajthanaki
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a><br>
Thank you for your response. I have cross checked the PyTorch version</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
1.5.0+cu92
</code></pre>
<p>I’m using fairseq-master version with Pytorch 1.5. This same issue is also happening fairseq-0.9 version</p>

username: ptrblck
cooked: <p>I cannot reproduce this issue with PyTorch alone.<br>
Could you run this code and check, if you are also seeing this issue without <code>fairseq</code>?</p>
<pre><code class="lang-python">import torch
d = torch.utils.data.IterableDataset
</code></pre>

username: jalajthanaki
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="83227">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">import torch
d = torch.utils.data.IterableDataset
</code></pre>
</blockquote>
</aside>
<p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you so much for your reply. I can able to run the given piece of code. Even I cannot see this issue without fairseq. I think then this is much more relevant to fairseq  then and raising the issue there again.</p>
<p>Your support and time is much appreciated. Thank you again <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Could you check, if the installation of <code>fairseq</code> is downgrading PyTorch somehow?</p>
Merge CNN model and Bag of Words

username: Erhan
cooked: <p>I hava a dataset consist of images and corresponding titles.<br>
This is a classification problem and there are 30 classes.<br>
I want to create CNN model with images and Bag of Words model with titles.<br>
I created vocubulary with text data and got output vectors for each titles from Bag of Words.<br>
I just want to merge CNN model and Bag of Words output to get one output.</p>
<p>Actually, I wanted to concatenate vectors from Bag of Words to first CNN fully connected layer.<br>
But vectors includes just frequencies numbers and merging this in the fc layer doesn’t make sense.</p>
<p>Is there any advice ?</p>
Quantized MaxPool2d and AdaptiveAvgPool2d

username: JC_DL
cooked: <p>Hi all,</p>
<p>I have been experimenting with the post static quantization feature on VGG-16.<br>
I know that <code>torch.quantization.convert()</code> will automatically remap every layer in the model to its quantized implementation.</p>
<p>However, i noticed that, a few types of layer is not converted, which is:</p>
<blockquote>
<p><code>nn.MaxPool2d()</code> , <code>nn.AdaptiveAvgPool2d()</code> and <code>nn.Dropout()</code></p>
</blockquote>
<p>I believe <code>nn.Dropout()</code> should not be an issue, whether its quantized or not.<br>
However, i am not sure if <code>nn.MaxPool2d()</code> and  <code>nn.AdaptiveAvgPool2d()</code> would do any difference if it is not quantized.</p>
<p>I have seen <code>nn.quantized.MaxPool2d()</code> being mentioned <a href="https://pytorch.org/docs/stable/quantization.html#torch-nn-quantized" rel="noopener nofollow ugc">here</a> and tried to remap my layer to this module. But, it seems like it is still referring to <code>nn.modules.pooling.MaxPool2d()</code> when i check the layer type after reassigning.</p>
<p>I have also seen <code>nn.quantized.functional.MaxPool2d()</code> and <code>nn.quantized.functional.AdaptiveAvgPool2d()</code> being mentioned in the Quantization <a href="https://pytorch.org/docs/stable/quantization.html#torch-nn-quantized-functional" rel="noopener nofollow ugc">documentation</a>. But i have read from the forum, and found that, it is not conventional to directly call <code>functional</code>, instead, its <code>module</code> or its wrapper class should be called.</p>
<p>So, i would like to ask, is there any effect to my quantized model performance if i don’t change the <code>nn.MaxPool2d()</code> and <code>nn.AdaptiveAvgPool2d()</code> to their quantized version?</p>
<p>Should i just leave <code>nn.MaxPool2d()</code> and <code>nn.AdaptiveAvgPool2d()</code> as it is?</p>
<p>Or, if i should change to their quantized implementation, how should i do it?</p>
<p>Thanks.</p>

username: raghuramank100
cooked: <p>You do not need to change MaxPool2d() and adaptiveAvgPool2d() from nn to nn.quantized. These operations do not require calibration and are automatically converted to quantized operations when convert is called. Under the hood, these modules call the appropriate function when quantized values are passed as input.</p>
Torch.epsilon()

username: fermat97
cooked: <p>Is there something similar to tf.keras.backend.epsilon() in pytorch? instead of explicitly define a very small number each time, we can use something like torch.epsilon() for instance.</p>

username: bhushans23
cooked: <p>No.<br>
AFAIK, PyTorch does not have anything similar epsilon as you mention above.<br>
IMO, it is good practice to mention our own epsilon value in the program and not depend on the framework.</p>

username: SimonW
cooked: <p>Just do something like</p>
<pre><code class="lang-auto">eps=1e-10
</code></pre>
<p>at some where in your codebase and use <code>eps</code> throughout… I don’t see why you would need one provided from pytorch core lib.</p>

username: rasbt
cooked: <aside class="quote no-group" data-username="fermat97" data-post="1" data-topic="30497">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/8edcca/48.png" class="avatar"> fermat97:</div>
<blockquote>
<p>tf.keras.backend.epsilon()</p>
</blockquote>
</aside>
<p>I agree that this would be overkill. It’s literally just</p>
<pre data-code-wrap="python"><code class="lang-python">@tf_export('keras.backend.epsilon')
def epsilon():
  """Returns the value of the fuzz factor used in numeric expressions.
  Returns:
      A float.
  """
  return _EPSILON
</code></pre>
<p>(Source: <a href="https://github.com/tensorflow/tensorflow/blob/r1.12/tensorflow/python/keras/backend.py" rel="noopener nofollow ugc">https://github.com/tensorflow/tensorflow/blob/r1.12/tensorflow/python/keras/backend.py</a>)<br>
where <code>_EPSILON = 1e-7</code>.</p>

username: fermat97
cooked: <p>Thanks, you are right.  I have done exactly something like <code>eps=1e-10</code>, I just thought maybe there is something  which I am not aware of.</p>

username: Rami_Nasser
cooked: <pre><code class="lang-auto">torch.finfo(torch.float32).eps
</code></pre>
How to integrate Multiheadattention with LSTM output

username: Talita
cooked: <p>Say that we use the LSTM example from the pytorch docs:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch.nn as nn 
&gt;&gt;&gt; rnn = nn.LSTM(10,20,2)
&gt;&gt;&gt; x = torch.randn(5,3,10)
&gt;&gt;&gt; output, (hn, cn) = rnn(x)
</code></pre>
<p>And that we feed the last hidden state to a MLP to get a prediction.</p>
<p>How can we feed the outputs from the LSTM to Multiheadattention ??</p>
Feature Map Upsampling

username: MUnique
cooked: <p>Hello,<br>
I have a tensor in size of (B<em>T</em>C<em>H</em>W) in which B is the batch size, T is the number of frames of the video, C the number of channels, and H,W the spatial size. This tensor is the extracted feature map of testing some videos on a pre-trained network. As the number of frames may vary from a video to another I wanted to interpolate all of the possible tensors to have the same temporal size.<br>
I tried:<br>
features_m = F.upsample(features_m, 128, mode=‘linear’)<br>
but it didn’t work. The size of the tensors only may vary in their second dimension. I want them to all have 128 timesteps. Would you guide me on how to do it?<br>
Thanks,</p>

username: ptrblck
cooked: <p>You could permute the dimensions and use the temporal dimension as a “fake” spatial dimension.<br>
Afterwards you could permute it back to the original shape.</p>
<p>Let me know, if this would work for you.</p>

username: MUnique
cooked: <p>Thanks, yes it works.</p>
Multi GPU training, Data parallel vs Apex ddp for Semantic segmentation

username: henry_Kang
cooked: <p>Hello. I am currently using the 2 GPU machines in the lab.<br>
The first one, it has 2 Titan V.<br>
Second, the other has 4 Titan V.<br>
When I train the same dataset but 2 times bigger batch size but 2 titan V result is better.<br>
The network is Mobilenet V3 based network.<br>
I have read some articles, and they said that Sync batch normalization could be helpful.<br>
I use the Apex version of Sync batch normalization.<br>
<code>Preformatted text</code>model = apex.parallel.convert_syncbn_model(model)<br>
However, still, 2 GPUs have a better result.<br>
So I am looking for another article. They said in some cases, the DDP is required if I use a lot of GPU.<br>
Does anyone have the same experience when you use multi GPU training?</p>
<p>The second thought is the multi-scale training, I read other published paper with code. They use multi-scale training with multiple GPUs. Does this effects the segmentation result?<br>
Thank you.</p>

username: ptrblck
cooked: <p>Did you play around with some hyperparameters, e.g. did you try to lower the learning rate for the multi-GPU setup?<br>
I assume your model doesn’t converge that well using 4 GPUs compared to the 2GPU run?</p>

username: henry_Kang
cooked: <p>I use the same hyper-parameters, same loss function, same model same initialization technique…<br>
My dataset as you remember that it is quite imbalanced data.<br>
For this reason, I use the Focal-Tversky loss(in my current experience it gives the best result).<br>
However, if I use more number of GPU should I reduce the lower learning rate?<br>
Could you teach me why I should reduce the learning rate?<br>
Thank you for your answer.</p>

username: ptrblck
cooked: <p>I’ve seen some experiments for large scale systems, where the learning rate was adapted to the batch size as seen in <a href="https://research.fb.com/wp-content/uploads/2017/06/imagenet1kin1h5.pdf">Training ImageNet in 1 hour</a>.<br>
However, this effect should be much smaller in your setup. Might still be worth a try to lower it and see, if it changes the convergence.</p>

username: henry_Kang
cooked: <p>Yes I will try. These days, I realize that experience is very important about this field for optimization.<br>
Thank you.</p>
Different learning rates for Adam gives Precision and Recall 0

username: Romina_Baila
cooked: <p>Hi there!<br>
I am fairly new to Pytorch and I am trying to provide a different learning rate for the parameters from BERT, and the rest of the model’s parameters to have the same lr.</p>
<p>My model class looks like this (it’s from a tutorial):</p>
<pre><code class="lang-auto">class BERTGRUModel(nn.Module):
    def __init__(self,
                 bert,
                 hidden_dim,
                 output_dim,
                 n_layers,
                 bidirectional,
                 dropout):
        
        super().__init__()
        
        self.bert = bert
        
        embedding_dim = bert.config.to_dict()['hidden_size']
        
        self.rnn = nn.GRU(embedding_dim,
                          hidden_dim,
                          num_layers = n_layers,
                          bidirectional = bidirectional,
                          batch_first = True,
                          dropout = 0 if n_layers &lt; 2 else dropout)
        
        self.dropout = nn.Dropout(dropout)
        
        self.out = nn.Linear(hidden_dim * 2 if bidirectional else hidden_dim, output_dim)
</code></pre>
<p>And I am trying to give the different learning rates like this:</p>
<pre><code class="lang-auto">optimizer = optim.Adam([{'params': model.rnn.parameters(), 'lr': 0.001},{'params': model.out.parameters(), 'lr': 0.001}, {'params': model.dropout.parameters(), 'lr': 0.001}, {'params': model.bert.parameters(), 'lr': 1e-5}])
</code></pre>
<p>But this performs worse than simply giving:</p>
<pre><code class="lang-auto">optimizer = optim.Adam(model.parameters())
</code></pre>
<p>It gives a worse accuracy and the precision and recall drop down to 0 (for each batch).</p>
<p>I’m sure I am doing something wrong, I just can’t figure out what.</p>

username: ptrblck
cooked: <p>The dropout layer doesn’t have trainable parameters, but besides that the code looks alright.<br>
Since your model is training with the “standard” approach, could you use your per-parameter optimizer with the same learning rate for all parameters, as you’ve used in the working approach?<br>
Assuming your model is training fine afterwards, the currently specified learning rates might be bad for your new approach.</p>

username: Romina_Baila
cooked: <p>You were right, thank you! It was the value I gave to the learning rate.</p>
Why x.grad = 70?

username: complexfilter
cooked: <p>import torch<br>
x = torch.tensor(5., requires_grad=True)<br>
y = 2<em>x</em>x<br>
x.data*=6<br>
y.backward()<br>
print(x.grad)</p>

username: ptrblck
cooked: <p>The manipulation of the <code>.data</code> attribute cannot be tracked by Autograd and can yield wrong results (as in this case) and other unexpected side effects.<br>
Remove the <code>x.data</code> manipulation and it should work. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>