akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Understanding PyTorch geometric add aggregation function during message passing

pizzaman: <p>Just getting started with Pytorch-geometric.</p>
<p>Let’s say I have an undirected graph, with four nodes, each with a single feature, and I wish to implement the Graph Convolutional layer as shown in the documentation here:</p>
<p><a href="https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch-geometric.readthedocs.io/en/latest/notes/create_gnn.html</a></p>
<p>Following those instructions, my message function would return a tensor<br>
norm.view(-1, 1) * x_j</p>
<p>Eventually I would like to recover a 4*1 tensor, where each entry is the normalised<br>
sum of the features of itself and neighbouring nodes, as per the equation shown in the tutorial.</p>
<p>Step 5 of the tutorial is performed by the add function. How exactly does it go about this?<br>
norm.view(-1, 1) * x_j  Has dimension (# of edges including self loops, <span class="hashtag">#of</span> features)</p>
<p>How does the add aggregation infer which edge belongs to which node in this case, so that it can<br>
perform the correct sum</p>
<p>What I believe happens is that the add function also gets passed the original edge index, which it can use to map the output of the message function, perhaps using torch_geometric.scatter, to create the correct sum</p>
Nested Sums for the input of a model

g33sean: <p>Hey all, say you have some model F(input), how would you do a nested summation in Pytorch such that you would sum over the input such that it wold sum(sum(F(input)-target) where the first sum is summing over the inputs and second sum is summing over the difference between F(input)-target</p>

ptrblck: <p>Could you post a small example using random tensors, which shows the desired output, please?</p>

g33sean: <p>Thank you for the response, say your model outputs x and your target values are y. So x and y will be<br>
x=torch.rand(100,1,32,32)<br>
y=torch.rand(100,1,32,32).<br>
So i want to sum over input x first before I sum over the difference between the pixels between x and y. I want the output to be a scalar, i think it would be something like this<br>
((x-y)**2).sum(dim=[1,2,3]).sum()</p>

ptrblck: <aside class="quote no-group" data-username="g33sean" data-post="3" data-topic="94242">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/g33sean/48/14962_2.png" class="avatar"> g33sean:</div>
<blockquote>
<p>So i want to sum over input x first before I sum over the difference between the pixels between x and y.</p>
</blockquote>
</aside>
<p>In your code snippet you would:</p>
<ul>
<li>calculate the difference between <code>x</code> and <code>y</code>: <code>(x-y)</code></li>
<li>square the difference: <code>()**2</code></li>
<li>apply the sum in <code>dim=[1,2,3]</code> so that your result should have the shape <code>[N]</code></li>
<li>sum over the batch dimension.</li>
</ul>
<p>You would get he same result, if you just apply the sum over all dimensions directly:</p>
<pre data-code-wrap="python"><code class="lang-python">((x-y)**2).sum()
</code></pre>
<p>If you want to apply the sum on <code>x</code> before, I assume you would like to use broadcasting afterwards to calculate the difference?<br>
If so, this code might work:</p>
<pre data-code-wrap="python"><code class="lang-python">((x.sum([1, 2, 3], keepdim=True) - y)**2).sum()
</code></pre>
<p>but please double check the result, as I’m still unsure if I understand the use case correctly.</p>
Torch=1.1.0 is not compatible with cuda 10.2

zahra: <p>Hi<br>
I need to have torch==1.1.0 and before I used</p>
<pre><code class="lang-auto"># CUDA 10.0
conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 -c pytorch
</code></pre>
<p>But now I want to install it again in another virtual environment and get compatibility error. This error says that my cuda version is 10.2.</p>
<p>Also, I tried</p>
<pre><code class="lang-auto">pip install torch==1.1.0 torchvision==0.3.0 -f https://download.pytorch.org/whl/cu90/torch_stable.html
</code></pre>
<p>but I recieved this error:</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch==1.1.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 1.4.0, 1.5.0, 1.5.1, 1.6.0)
ERROR: No matching distribution found for torch==1.1.0
</code></pre>
<p>I can not find the solution.<br>
Is there any suggestions?</p>
<p>Best Regards</p>

ptrblck: <p>I think the PyTorch <code>1.1.0</code> binaries were only built with CUDA9.0 and CUDA10.0 as seen <a href="https://pytorch.org/get-started/previous-versions/#v110">here</a> (and cpu of course).<br>
Why do you need this particular PyTorch version and would you be able to install the latest one (<code>1.6</code>) with CUDA10.2?</p>

zahra: <p>Thanks for your reply.<br>
I want to run an open source code. One of its requirements is torch==1.1.0.</p>
<p>What should I do now?</p>

ptrblck: <p>If you need to run version <code>1.1.0</code>, you would need to install it with CUDA9.0 or 10.0.<br>
Alternatively, if you need PyTorch <code>1.1.0</code> and CUDA10.2 you would need to build form source.</p>

zahra: <p>How should I downgrade my cuda version from 10.2 to 10?</p>
<p>Also, I could not understand what is the meaning of source in the second suggestion.</p>

ptrblck: <p>You can find the install commands in my link from the previous post.</p>

zahra: <p>I checked it out but I think it relates to install torch in different versions of cuda.<br>
I can not find anything about downgrading version of cuda, am I wrong?</p>

victorc25: <p>Yes, install the previous version of CUDA</p>

zahra: <p>Thanks.<br>
Please guide me how I do it.</p>

ptrblck: <p>Could you post the error message you are seeing after running:</p>
<pre><code class="lang-auto">conda install pytorch==1.1.0 torchvision==0.3.0 cudatoolkit=10.0 -c pytorch
</code></pre>
Visualising model using hooks returns unexpected results

Sassan_Owssia: <p>I am a pytorch beginner and tried to visualize the last convolution layer of the resnet using feed forward hooks. I tried to create a heatmap that shows what part of the image ,according to the resnet model, contained the most important features. However, the resulting heatmap is not at all what you might expect. I’m not sure if the code I have written is wrong, or is there something I don’t understand about the hidden layers or the hooks in pytorch.</p>
<pre><code class="lang-auto">img, label = next(iter(valid_dl))
img = img.to(device)

weights = []
def get_layer(module, input, output):
  weights.append(output)

model.layer4[2].conv1.register_forward_hook(get_layer)
out = model(img)
activated = weights[0][5]
mean_act = activated.mean(0)
</code></pre>
<p>The code for the heatmap is as follows:</p>
<pre><code class="lang-auto">fig , ax = plt.subplots()
ax.imshow(img[5].cpu().permute(2, 1, 0))
ax.imshow(mean_act.cpu().detach(), alpha = 0.5, extent=(0, 335, 335, 0), 
          interpolation = 'bilinear', cmap = 'magma' )
</code></pre>
<p>But the resulting Image is unclear:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/1/419a96d00710697f2bcbe8b20d815898b7e7e832.png" alt="download (1)" data-base62-sha1="9mmeXgl4eIaSa3jOHQIRTde0Dxo" width="475" height="470"></p>
<p>Here is the original image without the heatmap:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/e/9ec72af87b0772c5c6b5987f6bc550c6c5acd555.png" alt="download (2)" data-base62-sha1="mEC9DHFcdzTB9Qu2lpCQKGJvigl" width="475" height="469"></p>
<p>I first hooked out the last convolutional layer of an arbitrary image in the restnet, and then took the mean over the 512 channels and ended up with a tensor of size 11 by 11. From what I understand this tensor should contain information about how the model classified our image. The heat map above shows that the bottom part of the image contributed the most to the classification. As I mentioned I’m a beginner and I am not sure that if my reasoning is correct. I would appreciate some insight on this subject.</p>
AttributeError: 'NoneType' object has no attribute 'log_softmax'

Kaneda: <p>i’m getting this error from criterion, it says that <code>inputs</code> is none which is strange since code works for fully connected layer.</p>
<p>here is the full project <a href="https://github.com/kanedaaaa/Pytorch-MNIST-fully-connected-nns" rel="nofollow noopener">https://github.com/kanedaaaa/Pytorch-MNIST-fully-connected-nns</a></p>
<p>you can find cnn.py in Models folder and data/train/test in main.py</p>
<p>code snippets:</p>
<pre><code class="lang-auto">class CnnNet(nn.Module):
  def __init__(self):
    super(CnnNet, self).__init__()
    self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, padding=1)  
    self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, padding=1) 
    self.pool = nn.MaxPool2d(2,2)
    self.fc1 = nn.Linear(64*7*7, 128) 
    self.fc2 = nn.Linear(128, 10)
    self.dropout = torch.nn.Dropout(p=0.5)
    self.relu = torch.nn.ReLU()

  def forward(self, x):
    x = self.conv1(x)
    x = self.relu(x)
    x = self.pool(x)
    x = self.conv2(x) 
    x = self.relu(x)
    x = self.pool(x) 
    x = x.reshape(x.size(0), -1) 
    x = self.fc1(x) 
    x = self.dropout(x)

    prediction = self.fc2(x) 
</code></pre>
<pre><code class="lang-auto">def train():
	for epoch in range(num_epochs):
		running_loss = 0
		for i, (inputs, labels) in enumerate(trainloader):
			#inputs, labels = inputs.reshape(-1, 28*28).to(device), labels.to(device)
			inputs, labels = inputs.to(device), labels.to(device)

			optimizer.zero_grad()

			output= net(inputs)
			loss = criterion(output, labels)
			
			loss.backward()
			optimizer.step()

			running_loss =+ loss.item() * inputs.size(0)

			if(i+1) % 100 == 0:
				print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

		loss_values.append(running_loss / total_step)
</code></pre>
<p>btw, fully connected layer needs reshaped inputs.</p>

Nikronic: <p>Hi, you forgot to return <code>prediction</code>. Your <code>forward</code> returns nothing.</p>
<aside class="quote no-group" data-username="Kaneda" data-post="1" data-topic="94538">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaneda/48/24889_2.png" class="avatar"> Kaneda:</div>
<blockquote>
<p><code>prediction = self.fc2(x) </code></p>
</blockquote>
</aside>
<p>Bests</p>

Kaneda: <p>Moral of the story: Dont write code at 5am</p>
Prediction function for LSTM text generator gives 0 matches

grizzlycoder: <p>Hello,</p>
<p>I am working on a next word prediction model using a regularised LSTM architecture. I am struggling to understand why my model performs bad on inference. I have trained a LSTM model on the Penn Tree Bank dataset reaching about 94 in test perplexity, while this is not the best result I think it should give some correct predictions.</p>
<p>When I run my prediction functions on the test file, I get 0 matches however I do see predictions. I am not sure if exact matches are not supposed to happen or if my function itself has some error. Here is my code:</p>
<pre><code class="lang-auto">#Model Loading 

PATH = "./ptb_model.pth"
model = Model(vocab_size, hidden_size, layer_num, dropout, winit, lstm_type)
model = torch.load(PATH)

print("Vocab Size: {}".format(vocab_size))
unique_index_word = dict((i,c) for i,c in enumerate(sorted(vocab)))

def predict(string, model):
    batch_size = 1
    model.eval()
    string = clean_text(string)
    string = tokenize_text(string)
    string = to_lower_case_list(string)
    hidden = model.state_init(batch_size)
    try:
      for w in string:
          ix = torch.tensor([[unique_word_index[w]]])
          output, hidden= model(ix, hidden)
      _ , top_ix = torch.topk(output[0], k=10)
      choices = top_ix.tolist()
      choice = np.random.choice(choices[0])

      out_word = unique_index_word[choice]
    except:
      out_word  = "Error"
    return out_word

with open("ptb.test.txt", 'r') as f:
  lines = f.readlines()
  matches = 0
  total = 0
  for line in lines:
    line = line.strip(" ")
    text  = line.split(" ")
    if len(text) &gt; 7:
      #print(text)
      exact = text[7]
      text = text[:7]
      text = " ".join(text)
      out = predict(text, model)
      print("Sequence: {}".format(text))
      print("Prediction: {} Exact: {}".format(out, exact))
      print("*"*10)
      total += 1
      if out == exact:
        matches += 1
      
  print(matches/total)
</code></pre>
<p>I am also not sure how to include an accuracy measure in my training process. Here is my code for the perplexity:</p>
<pre><code class="lang-auto">def perplexity(data, model):
    with torch.no_grad():
        losses = []
        states = model.state_init(batch_size)
        accuracy = 0
        for x, y in data:
            scores, states = model(x, states)
            loss = nll_loss(scores, y)
            ps = torch.exp(scores)
            #print(y)
            #print(scores)
            #equality = (y.data[0] == ps.max(dim = 1))
            #accuracy += equality.type(torch.FloatTensor).mean()
            #print(accuracy)
            losses.append(loss.data.item()/batch_size)
    return np.exp(np.mean(losses)), np.mean(losses), accuracy

</code></pre>
<p>If I’m not wrong, since I have 0 matches on inference, wouldn’t my model accuracy be 0? This just seems a bit strange to me. If any one knows what’s wrong, I would be very grateful!</p>
VAE on pinwheel of 2D points

gwg: <p>I am trying to train a VAE to replicate Figure 1c in <a href="https://arxiv.org/abs/1603.06277" rel="noopener nofollow ugc">this paper</a>:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/4/4adb3fc8fa967ada183a373c81f2e332184ffc92.jpg" alt="22%20PM" data-base62-sha1="aGd4sFce6Ipltnp15QoevXMuDmO" width="171" height="203"></p>
<p>In other words, I want to train a VAE on 2D points that form a pinwheel (or really any weird shape) and then generate or sample 2D points from the trained VAE that are also in a pinwheel. I am using the <a href="https://github.com/pytorch/examples/blob/master/vae/main.py" rel="noopener nofollow ugc">example PyTorch VAE</a> with two modifications:</p>
<ol>
<li>
<p>All the layers are 2-dimensional:</p>
<pre data-code-wrap="python"><code class="lang-python">self.fc1  = nn.Linear(2, 2)
self.fc21 = nn.Linear(2, 2)
self.fc22 = nn.Linear(2, 2)
self.fc3  = nn.Linear(2, 2)
self.fc4  = nn.Linear(2, 2)
</code></pre>
</li>
<li>
<p>I changed the loss to <code>F.mse_loss</code> (also tried Euclidean distance) instead of <code>F.binary_cross_entropy</code> since I am dealing with cartesian points instead of black and white images.</p>
</li>
</ol>
<p>That said, I am unable to learn the pinwheel dataset. Here is what happens if I sample from the VAE 10000 times:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/9/96f6791a88f828d9761d8f71ee09d3738d2be593.png" alt="mse_loss" data-base62-sha1="lxtGzxBhaXpXoTXZvW5j5Agj3sn" width="390" height="252"></p>
<p>I’ve tried both Adam and SGD with lr=0.001. Any idea what is going wrong?</p>

always: <p>I have a similar problem, have you found a solution by now?</p>

gwg: <p>It’s been a while, but I think I was able to get this work by mimicking the architecture in the paper, which is roughly linear &gt; tanh &gt; linear &gt; tanh. See the definition here: <a href="https://github.com/mattjj/svae/blob/master/experiments/gmm_svae_synth.py" rel="nofollow noopener">https://github.com/mattjj/svae/blob/master/experiments/gmm_svae_synth.py</a>.</p>
Difference in result for sci-kit learn PCA and manual PCA in pytorch

Mah: <p>I’m trying to get principal components via two different methods but I get different results.<br>
Can you show me what I am missing.<br>
Thanks.</p>
<p>B1 = np.array([<br>
[0.387,4878, 5.42],<br>
[0.723,12104,5.25],<br>
[1,12756,5.52],<br>
[1.524,6787,3.94],<br>
[2.3,4.1,3.21],<br>
[3.1,3.2,2.6]])<br>
from sklearn.decomposition import PCA<br>
pca = PCA(2)<br>
pca.fit(B1)<br>
pca1=pca.fit_transform(B1)<br>
print(pca1)</p>
<p>[[-1.21071628e+03  1.84225231e+00]<br>
[ 6.01528345e+03 -1.61508354e-01]<br>
[ 6.66728345e+03 -3.15444437e-01]<br>
[ 6.98283241e+02 -4.42740940e-01]<br>
[-6.08461682e+03  3.72122654e-02]<br>
[-6.08551704e+03 -9.59770842e-01]]</p>
<p>A = torch.tensor(B1, dtype=torch.float)<br>
B = torch.tensor(B1, dtype=torch.float)<br>
xm = torch.mean(torch.transpose(B,0,1), 1, keepdim=True)<br>
xm=torch.transpose(xm,0,1)<br>
c=torch.zeros(B.shape)<br>
c = A - xm[0]<br>
lenn = c.shape[0]<br>
covariance = torch.matmul(torch.transpose(B, 0, 1),B)/(lenn-1)<br>
es, vs = torch.symeig(torch.transpose(covariance, 0, 1), eigenvectors=True)</p>
<p>len_pca=2<br>
pc1=torch.matmul(torch.transpose(vs, 0, 1),torch.transpose(c, 0, 1))</p>
<p>pca1=torch.transpose(pca1, 0, 1)<br>
print(pca1[:,:lenC])</p>
<p>tensor([[-1.9730e+00, -2.6653e-03],<br>
[ 1.0763e+00,  2.1804e+00],<br>
[ 1.2333e+00,  2.0192e+00],<br>
[ 6.0225e-01,  4.1150e-01],<br>
[-9.3635e-01, -2.1175e+00],<br>
[-2.4548e-03, -2.4910e+00]])</p>
How to implement class weights for MSE loss?

hadaev8: <p>Cant find in google anything about it.</p>

KFrank: <p>Hi Had!</p>
<aside class="quote no-group quote-modified" data-username="hadaev8" data-post="1" data-topic="94428" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hadaev8/48/16280_2.png" class="avatar"> hadaev8:</div>
<blockquote>
<p>How to implement class weights for MSE loss?</p>
</blockquote>
</aside>
<p>The short answer is that MSE loss is not a loss that would naturally<br>
be used with classes.  As such, pytorch’s <code>MSELoss</code> does not attempt<br>
to implement any feature that might be considered analogous to<br>
“class weights.”</p>
<p>The reason is that MSE loss is naturally a measure how much two<br>
sets of real numbers differ from one another.</p>
<p>Class labels (even though implemented in pytorch as integer<br>
categorical class labels) aren’t conceptually really numbers.  You<br>
can test whether two class labels are the same, “bird” = “bird”,<br>
but you can’t ask how far apart two class labels are.  That is, what<br>
number would you assign to “bird” - “fish”?  Is “bird” a better incorrect<br>
label for “fish” than is “reptile”?</p>
<p>Having said all that, you can write your own MSE loss that takes<br>
sample weights.  If you have some scheme for assigning samples<br>
to classes, you could then construct a vector of sample weights<br>
(for each batch of samples) that are given by the class weight of<br>
each sample’s corresponding class.</p>
<p>Good luck.</p>
<p>K. Frank</p>

hadaev8: <p>Well, in my case mse seems to be better.<br>
This is not simple classification task, i use it in the middle of seq2seq model.<br>
But seems like sample weight is only way.</p>
"index out of bounds" when moving to device

mordith: <p>Hi,</p>
<p>I get the following cuda error when I try to move a tensor from cpu to device:</p>
<blockquote>
<p>/opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda <a></a>-&gt;auto::operator()(int)-&gt;auto: block: [0,0,0], thread: [0,0,0] Assertion <code>index &gt;= -sizes[i] &amp;&amp; index &lt; sizes[i] &amp;&amp; "index out of bounds"</code> failed.<br>
CUDA error: device-side assert triggered</p>
</blockquote>
<p>The code I am using is as follows:</p>
<pre><code class="lang-auto">cur_inds = has_answer_vector[sample_ind]
tensorized_inds = torch.from_numpy(np.array(cur_inds))
cur_positive_inds = tensorized_inds.to(logits.device).long()
</code></pre>
<p>The error occurs on the final line, but it only fails on some batches and not others.</p>
<p><code>has_answer_vector</code> is simply a list of lists with an integer in them<br>
<code>logits</code> is a FloatTensor</p>
<p>An example of the values I get when it fails are:</p>
<pre><code class="lang-auto">has_answer_vector: [[1], [1], [1], [1], [0], [0], [0], [0], [16], [0], [1], [2], [146], [0], [9], [3]]
sample ind: 13
cur_inds: [0]
tensorized_inds: tensor([0])
tensorized_inds.device : cpu
logits.device : cuda:0
</code></pre>
<p>Any ideas why this happens?</p>

ptrblck: <p>CUDA operations are executed asynchronously, so a failed operation might raise an assertion, which will be triggered by the next CUDA operation.<br>
In your case an indexing failed and you see the error in the next <code>.to()</code> operation.<br>
You can rerun the code via:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>to get the proper line of code which is failing (or run the code on the CPU alternatively).</p>
Convert a JpegImageFile to a tensor to train a ResNet

DeepLearner17: <p>Hello,<br>
l have a jpeg image of (3,224,244). l need to put it in a variable image but it needs to be convert to a tensor (1,3,244,224) to train a Resnet152.<br>
l did the following :</p>
<p>from PIL import Image<br>
img_path="/data/v_PlayingSitar_g25_c04_0256.jpeg"<br>
<span class="hashtag">#image</span> = torch.Tensor(1,3, 224, 224).normal_()<br>
image = Image.open(img_path)<br>
image = Variable(image)</p>
<p>l got the following error :<br>
<strong>RuntimeError: Variable data has to be a tensor, but got JpegImageFile</strong></p>
<p>Thank you</p>

jytug: <p>You need to convert the <code>PIL</code> image to <code>torch.Tensor</code>. You can do this using <code>torchvision.transforms.ToTensor</code>, like this:</p>
<pre><code class="lang-auto">from PIL import Image
from torchvision.transforms import ToTensor

img_path="/data/v_PlayingSitar_g25_c04_0256.jpeg"
image = Image.open(img_path)
image = ToTensor()(image).unsqueeze(0) # unsqueeze to add artificial first dimension
image = Variable(image)
</code></pre>
<p>You can find much more information in the tutorials:<br>
<a href="http://pytorch.org/tutorials/" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/</a><br>
Here’s one specifically about data loading:<br>
<a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#sphx-glr-beginner-data-loading-tutorial-py" class="onebox" target="_blank" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/data_loading_tutorial.html#sphx-glr-beginner-data-loading-tutorial-py</a></p>

DeepLearner17: <p>Thank you  <a class="mention" href="/u/jytug">@jytug</a>, l got the following error after running</p>
<pre><code>image = ToTensor(image).unsqueeze(0)
TypeError: object() takes no parameters</code></pre>

jytug: <p>I have edited the answer, it should be correct now</p>

Dunfash: <p>I got<br>
TypeError: invalid shape (3,224,224) for image data</p>

Dunfash: <p>from PIL import Image<br>
import cv2<br>
for ind in labels[labels[‘LABEL’]==1][‘filename’][-5:]:<br>
path = ind<br>
imgr = cv2.imread(path)<br>
imgr = cv2.resize(imgr,dsize=(224,224))<br>
trans = transforms.ToTensor()<br>
print(imgr.shape) <span class="hashtag">#prints</span>(224,224,3)<br>
plt.imshow(trans(imgr)) <span class="hashtag">#displayes</span> error** invalid shape (3,224,224) for image data</p>

ptrblck: <p><code>matplotlib.pyplot.imshow</code> expects a numpy array as <code>[height, width, channels]</code>, so you would have to permute the transformed tensor and transform it back to a numpy array via:</p>
<pre><code class="lang-python">x = trans(imgr)
x = x.permute(1, 2, 0).numpy()
plt.imshow(x)
</code></pre>
Is it possible to use YOLO without training?

Jacek_Poplawski: <p>Hello,</p>
<p>I have experience with ML and DL (competitions on Kaggle), however I never tried to use YOLO.</p>
<p>I am just doing some research and looks like in torchvision.models there are no models which can be used with YOLO, am I correct?</p>
<p>I found some articles/tutorials how to implement YOLO in PyTorch but they contain training. Does it mean there is no way to divide image into rectangles with objects (like cars, people, etc) without having data to train first?</p>
<p>I want to build system to recognize specific objects on the images from camera. I want to use data which contains only 0/1 labels, so no bounding boxes.</p>
<p>My idea was to divide full images from camera into smaller images and use only these smaller images for training, but for that I need pretrained YOLO model.</p>

Lin_Jia: <p>YOLO contains the training stage to identify certain objects. I bet there are some pretrained YOLO models out there. You can use them as long as the specific objects you are interested in are the same as the pretrained YOLO model. What is your end goal, do you expect the model to output “YES/NO” on certain objects from camera feed? BTW, if the camera is on iPhone, feel free to check out Apple’s native support. I think they already have a framework to help you detect basic objects, such as a computer, from camera feed: <a href="https://developer.apple.com/documentation/vision" rel="nofollow noopener">https://developer.apple.com/documentation/vision</a>. Maybe Google has a similar framework on Android.</p>

Lin_Jia: <p>Also, if your training data is “YES/NO” answer on whether or not certain objects exist in camera feed, and if your trained model only need to output “YES/NO”. I don’t think you have to use YOLO. You can just have a VGG, or AlexNet, but the output can be a 1D vector, where each value represent the probability of certain object exist or not. I high recommend checking out data argument tricks (such as training data rotation, zoom in/out) for this kind of training task.</p>

Jacek_Poplawski: <p>My idea was to automatically split full size Images from camera and then to manually label cropped Images.<br>
Then during training to split images again and train second model on cropped Images and labels.<br>
Then during test to split new Images with pretrained YOLO model and predict cropped Images with trained model.</p>
<p>My question was about PyTorch models to use, is there any pretrained model I could use for such task or it is not possible.</p>
<p>As I said I am quite experienced with training DL (you can check my Kaggle profile) what I dont know yet is YOLO simply because I never had chance to use such data.</p>
How to go into the codes behind the '_backend'

tong: <p>Hi, all. I am new for pytorch. I wonder how to read the deeper code behind the ‘_backend’ in the module. The example codes are in the file “torch/nn/modules/sparse.py”, around line 96:<br>
"<br>
def forward(self, input):<br>
padding_idx = self.padding_idx<br>
if padding_idx is None:<br>
padding_idx = -1<br>
return self._backend.Embedding.apply(<br>
input, self.weight,<br>
padding_idx, self.max_norm, self.norm_type,<br>
self.scale_grad_by_freq, self.sparse<br>
)<br>
"<br>
Here, I wonder how to read the code behind “self._backend.Embedding.apply”? The “_backend” is actually ‘’ thnn", and where can I find the corresponding codes to the ‘Embedding.apply’ Thanks for help.</p>

Will_Zhang: <p>Got the same issue, with v0.4 I got a <code>raise NotImplementedError</code> for a line that called  <code>Embedding._backend.Embedding.apply()</code>. How should I interpret this?</p>

sohamg: <p>I think you can use <code>torch.nn.functional</code> and use <code>F.embedding</code> in v0.4</p>

onbigion13: <p>I’ve gotten the same trouble as you did, did you solve your problem? It raise  an erroe that _backend is not implemented, and I don’t know how to implement nn.mbedding to pass to inputs.</p>
Pytorch DataLoader extremely slow first epoch

sunsunsun: <p>When I create pytorch DataLoader and start iterating – I got extremely slow first epoch (x10–x30 slower then all next epochs). Moreover, this problem occurs only with train dataset from google landmark recognition 2020 from Kaggle, I can’t reproduce this on synthetic images, also, tried to create folder with 500k images from GLR2020, everything worked well. Found few similar problems in pytorch forum without any solutions.</p>
<pre><code class="lang-auto">import argparse
import pandas as pd
import numpy as np
import os, sys
import multiprocessing, ray
import time
import cv2
import logging
import albumentations as albu
from torch.utils.data import Dataset, DataLoader

samples = 50000 # count of samples to speed up test
bs = 64 # batch size
dir = '/hdd0/datasets/ggl_landmark_recognition_2020/train' # directory with train data
all_files = pd.read_csv('/hdd0/datasets/ggl_landmark_recognition_2020/train.csv')
files = np.random.choice(all_files.id.values, 50000)
files = [os.path.join(_[0], _[1], _[2], _+'.jpg') for _ in files]

# augmentations
aug =  albu.Compose([albu.Resize(400, 400),
        albu.Rotate(limit=15),
        albu.ChannelDropout(p=0.1),
        albu.Normalize(),])

class ImgDataset:
    def __init__(self, path, files, augmentation = None):
        self.path = path
        self.files = {k:v for k, v in enumerate(files)}
        self.augmentation = augmentation

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        img_name = self.files[idx]
        img = np.array(cv2.imread(os.path.join(self.path, img_name)))
        if self.augmentation is not None:
            return self.augmentation(image=img)['image']


dtset = ImgDataset(dir,files, aug)
torchloader = DataLoader(dataset= dtset, batch_size=64, num_worker=16, shuffle=True)
for _ in range(3):
   t1 = time.time()
   for idx, val in enumerate(torchloader):
       pass
   t2 = time.time()
   print(str(t2-t1) +' sec')
</code></pre>
<p>Here are some examples of execution speed with different  <code>num_workers</code>  in DataLoader</p>
<pre><code class="lang-auto">#num_workers=0
273.1584792137146 sec
83.15653467178345 sec
83.67923021316528 sec

# num_workers = 8 
165.62366938591003 sec
10.405716896057129 sec
10.495309114456177 sec

# num_workers = 16
156.60744667053223 sec
8.051618099212646 sec
7.922858238220215 sec
</code></pre>
<p>Looks like problem is not with DataLoader, but with dataset. When I delete and reinitialise DataLoader object after first “long” iteration, everything still works fine. When I reinitialise dataset – long first iteration appears again. Moreover, I tracked my cpu utilisation via  <code>htop</code>  during this epochs with  <code>num_workers</code>  setted to 32, during first epoch utilisation is really low, only 1-2 of 32 cores are working, during other epochs ~all cores are working</p>
<p>Absolutely have no idea how to fix this, helps plz</p>
How to interleave two tensors along certain dimension?

tedli: <p>Hi there,</p>
<p>Say if I got two tensors like [[1,1],[1,1]] and [[2,2],[2,2]], how could I interleave them along n_w or n_h dimension to get [[1,2,1,2],[1,2,1,2]] or [[1,1],[2,2],[1,1],[2,2]]? In TensorFlow I could achieve such goal using tf.reshape after tf.stack, but transferring to PyTorch using view after torch.stack does not return what I want.</p>
<p>Thanks.</p>

ptrblck: <p>I’m sure there is a better way to achieve this, but as of now, you could do the following:</p>
<pre><code>a = torch.Tensor([[1,1], [1,1]])
b = torch.Tensor([[2,2], [2,2]])

torch.stack((a, b), dim=0).view(2, 4).t().contiguous().view(2, 4)
&gt;&gt; 1 2 1 2
&gt;&gt; 1 2 1 2
&gt;&gt; [torch.FloatTensor of size 2x4]

torch.stack((a, b), dim=0).view(4, 2).t().contiguous().view(4, 2)
&gt;&gt; 1 1
&gt;&gt; 2 2
&gt;&gt; 1 1 
&gt;&gt; 2 2
&gt;&gt; [torch.FloatTensor of size 4x2]</code></pre>

tedli: <p>I’m not sure if there is a better way, but your answer definitely saved my day. <img src="//discuss.pytorch.org/images/emoji/apple/grinning.png?v=5" title=":grinning:" class="emoji" alt=":grinning:"></p>
<p>Cheers!</p>

fbcotter: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="11332">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>torch.stack((a, b), dim=0).view(2, 4).t().contiguous().view(2, 4)</p>
</blockquote>
</aside>
<p>How come you don’t do</p>
<pre><code class="lang-auto">torch.stack((a,b), dim=2).view(2,4)
&gt;&gt; tensor([[ 1.,  2.,  1.,  2.],
           [ 1.,  2.,  1.,  2.]])
</code></pre>
<p>Similarly, the vertical interleave can be:</p>
<pre><code class="lang-auto">torch.stack((a,b), dim=1).view(4,2)
&gt;&gt; tensor([[ 1.,  1.],
           [ 2.,  2.],
           [ 1.,  1.],
           [ 2.,  2.]])
</code></pre>
<p>I just started using pytorch on version 0.4 so maybe it’s something they added recently. Also, I’m sure there’s a good reason you’ve made calls to <code>contiguous()</code>, is there something I should know about the view?</p>

ptrblck: <p>That’s what I mean by “I’m sure there is a better way to achieve this”. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=5" title=":wink:" class="emoji" alt=":wink:"><br>
Thanks for the code. It’s a lot clearer than my approach.</p>
<p>I’ve called <code>contiguous</code> on the result, since I transposed it with <code>t()</code>. This might lead to errors for some operations that require contiguous tensors:</p>
<pre><code class="lang-auto">a = torch.randn(2, 4)
print(a.is_contiguous())
print(a.t().is_contiguous())
</code></pre>

arogozhnikov: <pre><code class="lang-auto">from einops import rearrange
rearrange([a, b], 't h w -&gt; h (w t)')  # [[1,2,1,2],[1,2,1,2]]
rearrange([a, b], 't h w -&gt; (h t) w')  # [[1,1],[2,2],[1,1],[2,2]]
</code></pre>
<p>Same code in tensorflow and numpy</p>
Overfitting problem

lifeblack: <p>Hi! I wrote a network an I want to check it. I use one batch to see if it overfit or not… I plot loss during training … is this right?</p>
<pre><code class="lang-auto">model.to(device)
optimizer=torch.optim.SGD(model.parameters(),lr=0.00001)
Loss=np.array([])
for epoc in range(50):
    print("***********",epoc)
    for iteration,(x,r) in enumerate(Train_Loader):
      optimizer.zero_grad()
      y=Block(x,PHI)
      output,out = model(r.to(device),y.to(device),PHI)
      loss = my_loss(output.to(device), y.to(device),1,out.to(device))
      Loss=np.append(Loss,[loss])
      loss.backward()
      optimizer.step()
      end=time.time()
      period=end-start
      print('period', period)
      start=time.time()
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db6e57592e37800718db7d564801a23be1b9195a.png" alt="image" data-base62-sha1="vjaYcp2RKiv6W61pGn88fxY4eVk" width="506" height="343"><br>
final loss is around 100000… I think its wrong…</p>

Lin_Jia: <p>Correct me if I am wrong here. I think overfitting means that your model output a small loss on training data, but once you migrate to testing data, the loss is much higher. This means that your model learns a lot of details inside training data, but it is “over” fit for general cases, such as testing data.</p>
Undefined reference to `vmdLog2

xpan: <p>I got  “undefined reference to <code>vmdLog2'" and "undefined reference to </code>vmsLog2’” errors when I compile the source code of PyTorch. Has anyone ever met the same problem? How to solve this problem？</p>

tom: <p>These are from MKL, so likely something is up with your MKL installation.</p>
<p>From the <a href="https://github.com/pytorch/pytorch/issues?q=is%3Aissue+vmdlog2" rel="nofollow noopener">archived issues people have filed</a>, updating mkl (and for conda mkl-include) has helped people solve the issue.</p>
<p>Best regards</p>
<p>Thomas</p>

xpan: <p>Thanks for your kind help. I have solved this issue.</p>
<p>Many thanks and best regards.</p>
<p>Xiao</p>
NMS and RoiAlign Coordinate Types

kleingeo: <p>I am trying to implement 3D versions of NMS and RoiAlign for a 3D image detection network by taking some C++ and CUDA code from different sources and modifying them for 3D image volumes with BCHWD sizes. One thing I am curious about (and it isn’t clear in the documentation) is if the current NMS and RoiAlign code are meant for normalized or un-normalized coordinates for their box inputs. This won’t really be an issue for the NMS (since the intersection and union calculations would both be normalized) but may have an impact on the RoiAlign fuction.</p>
Cannot load model from disk

Thabang_Lukhetho: <p>I get this error when I load a model from disk. Pytorch version 1.4</p>
<pre><code class="lang-auto">ImportError: cannot import name 'Optional' from 'torch.jit.annotations'
</code></pre>

ptrblck: <p>This error is usually raised, if you are using a newer <code>torchvision</code> version with an older PyTorch version, which doesn’t support the needed utilities in the <code>torchvision</code> model.<br>
Make sure to update both libraries to their latest stable release (or matching versions, if you need an older one).</p>

Thabang_Lukhetho: <p>Thanks for the reply.</p>
How to get non-negative integer predictions from a regression model?

stark: <p>I’m using a pretrained VGG19 and fine-tuning (<em>freezing the conv layers</em>) its classifier parts (<em>i.e. 3x linear layers</em>).</p>
<p><code>loss = torch.nn.MSELoss(output, target)</code></p>
<p>where</p>
<p><code>output  = [13.7210, 1.6992, -0.1286, -0.9545, -0.9148, 2.3547]</code>, and<br>
<code>target  = [ 0., 0., 0., 0., 14., 1.])</code> <em>(each element is a count of respective class)</em></p>
<p>The calculated loss is <code>169.3941</code> that is completely useless since overall loss tends to increase as the model sees more and more images. Why I’m not getting the predictions closer to the targets?</p>

ptrblck: <p>If your model training is diverging, you could use “bad” hyperparameters such as a high learning rate or there might be a bug in your code (such as forgetting to zero out the gradients).<br>
As quick test I would recommend to overfit a small data sample (e.g. just 10 samples) and make sure your model is able to fit them perfectly.</p>

googlebot: <p>You probably need <a href="https://pytorch.org/docs/stable/generated/torch.nn.PoissonNLLLoss.html#torch.nn.PoissonNLLLoss" rel="nofollow noopener">PoissonNLLLoss</a></p>
DCGAN ReLU vs. Leaky ReLU

David_Refaeli: <p>I noticed that in <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" rel="nofollow noopener">DCGAN implementation</a> Generator has ReLU but Discriminator has leaky ReLU - any reason for the difference?</p>
<p>Also - anyone knows why the Discriminator 1st layer doesn’t have BN ?</p>

mariosasko: <p>Please read the <a href="https://arxiv.org/abs/1511.06434" rel="nofollow noopener">original paper</a> to understand the architecture choices. Most of them are decided empirically.</p>

vitouphy: <p>Training with GAN is very unstable. And I don’t think there is legit reason behind each choice. As <a class="mention" href="/u/mariosasko">@mariosasko</a> mentioned, those are empirically picked after many experiments (hyper-parameters tuning).</p>
Double Backpropagation in Pytorch

dr-faustus: <p>Hey guys!</p>
<p>Does anybody know how to implement double backpropagation in Pytorch? I’m specifically referring to the LeCun paper “Improving generalization performance using double backpropagation” (and it’s recent use for adversarial training in “Improving the Adversarial Robustness and Interpretability of Deep Neural Networks by Regularizing Their Input Gradients”).<br>
The idea is to use the square norm of the gradient of the criterion with respect to the input as a regularization term to decrease the rate of the change of the weights.<br>
Given that we would have to call the backward function of the criterion to have access to the gradients of the input, I guess there’s no easy way of doing it in one shot like in the papers.</p>

Jean-Eric_Campagne: <p>Hi, by any chance have found a Pytorch implementation? (Pytorch &gt;= 1.3.1)<br>
Thanks</p>

Shuan_Holmes: <p>I found a similar question <a href="https://discuss.pytorch.org/t/how-to-backward-the-derivative/17662/2">here</a> and another <a href="http://luizgh.github.io/libraries/2018/06/22/pytorch-doublebackprop/" rel="nofollow noopener">blog</a>.</p>
LSTMCell Bias question

Alan_Kong: <p>I read the documentation for LSTM cell:([<a href="https://pytorch.org/docs/master/generated/torch.nn.LSTMCell.html" rel="nofollow noopener">https://pytorch.org/docs/master/generated/torch.nn.LSTMCell.html</a>]</p>
<p>Noted that there are two biases required, namely:</p>
<ul>
<li>
<strong>~LSTMCell.bias_ih</strong>  – the learnable input-hidden bias, of shape (4*hidden_size)</li>
<li>
<strong>~LSTMCell.bias_hh</strong>  – the learnable hidden-hidden bias, of shape (4*hidden_size)</li>
</ul>
<p>Assuming that batch size is 1,  and the size of input(x) and hidden(h) state is 128 and 64, then:</p>
<p>i(t) = [1, 128]<br>
h(t) = [1, 64]</p>
<p>w(i) = [64, 128+64]<br>
b(i) = [64]</p>
<p>I feel uncertain why we need two biases, to fit each set of activation (i.e. sigmoid and tanh) functions.  As the shape of outputs must be equal to the hidden state. So one bias is enough for each set. why we need two biases for each set?</p>
<p>Also, for LSTM model, we have five activation functions, namely:</p>
<ol>
<li>forget gate (sigmoid)</li>
<li>input gate (sigmoid)</li>
<li>input gate (tanh)</li>
<li>output gate (sigmoid)</li>
<li>output gate (tanh)</li>
</ol>
<p>The documentation mentions 4*, instead of 5*.  Is it related to <span class="hashtag">#1</span> to <span class="hashtag">#4</span> above as <span class="hashtag">#5</span> is not required for bias due to solely c(t).</p>
<p>Thanks for your patience for reading.</p>

Alan_Kong: <p>I guess the probable reason is:</p>
<p>output of activation function = activation function (i.e. sigmoid/tanh) (matrix addition (w(i) * i(t) + b(i), w(h) * h(t-1) + b(h)))</p>
<p>which is different from:</p>
<p>output of activation function = activation function (i.e. sigmoid/tanh) (w(i + t-1) * matrix concatenation (i(t), h(t-1)) + b(h))) [as referred by the formulas from <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/" rel="nofollow noopener">colah’s blog</a></p>

mariosasko: <p>It’s for consistency reasons (see <a href="https://discuss.pytorch.org/t/why-rnn-needs-two-biases/557/4">this</a>).</p>
Why is it a bad idea to use python's `concurrent.futures` with pytorch and how can I paralellize batch loading in RL?

FabianSchuetze: <p>I am trying to load a batch from a replay buffer with pytorch asyncronously while optimizing the model parameters and thereby hide the batch loading latency. The program I run is as follows:</p>
<pre><code class="lang-python">for _ in range(100):
    begin = time.time()
    batch = sample_batch()
    batch_load += time.time() - begin
    begin = time.time()
    optimize(batch)
    optimize_time += time.time() - begin
</code></pre>
<p>When running this script, <code>batch_load</code> takes about 0.001 seconds and <code>optimize_time</code> about 0.009 seconds. To hide the latency of the batch_load (although it doesn’t take long in this program, it takes more time in another program which I would actually like to optimize), I thought I can use python<code>s </code>concurrent.futures<code>module to acquire a</code>future<code>from</code>sample_batch<code>and load it whilst</code>optimize` is running. This program instead looks as follows:</p>
<pre><code class="lang-auto">with concurrent.futures.ProcessPoolExecutor(max_workers=12) as executor:
    for _ in range(100):
        begin = time.time()
        future = executor.submit(sample_batch)
        batch_load += time.time() - begin
        begin = time.time()
        optimize(batch)
        optimize_time += time.time() - begin
        batch = future.result()
</code></pre>
<p>This turned out to be a pretty bad idea. The data loading time increases to 0.085 seconds and the optimization time increases to 0.13 seconds.</p>
<p>Can somebody kindly educate me on why the second program is so much slower than the first? Furthermore, does somebody have any ideas on how to hide data loading latency? I appreciate any answers and suggestions very much!</p>

mrshenli: <p>As <code>batch_load</code> measures the latency of <code>executor.submit</code>, I assume that’s the overhead of  <code>ProcessPoolExecutor</code>?</p>
<p>But it is still weird that the <code>optimize()</code> also increased a lot. Does <code>optimize()</code> run ops on GPU? If yes, you will need to either <code>torch.cuda.synchronize()</code> on that GPU, or use <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="nofollow noopener"><code>elapsed_time</code></a> to measure the latency. Because, CUDA ops returns when the op is inserted to the stream instead of when the op is done.</p>

FabianSchuetze: <p>Thank you <a class="mention" href="/u/mrshenli">@mrshenli</a> for your answer!</p>
<p>Indeed, the slower run time was caused entirely by the overhead of the <code>ProcessPoolExecutor</code>.  It is interesting that this context has implications also for non-asynchronous procedure calls. I measured the entire program again with longer-running tasks and the overhead of the  <code>ProcessPoolExecutor</code> seemed to be constant but the latency of data loading could be hid below the <code>optimize</code> call.</p>
<p>Again, thank you for your reply - It helped me a lot!</p>
Early stopping - Do I get the best model or the latest model?

GuillaumeF: <p>If the fitting of my neural network ends early because of early stopping, does Pytorch return the best model or the latest model fitted?</p>

ptrblck: <p>Usually you would stop the training and restore the “best” model, but it depends how you’ve implemented the early stopping.<br>
Could you share the code you are using, so that we could have a look? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Is there a function to evaluate mAP(mean average prescision) for object detection in torch or torchvision?

Flicic_Suo: <p>Just if there’s a guaranteed code I don’t have to implement and debug it myself <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
A question about the torch.histc function

alan_ayu: <p>I found histc function in new pytorch version and I want to use it to implement my some ideas. but I get a problem that I can only get the statistical result using this function. However, can I get inverse mapping from this result to the input tensor, for example, if I input a tensor with any possible size to torch.histc, how can I get the corresponding frequency and bin index of one tensor value to the histogram result ?<br>
Can you share me with any interesting idea? Thank you ！</p>

ptrblck: <p>I’m not sure if this is currently possible with this method.<br>
However, you could calculate the bin borders and use these values as a threshold to index your tensor.</p>
<pre><code class="lang-python">x = torch.arange(10).float()
low = 5.
high = 7.
(low &lt;= x) * (x &lt; high)
</code></pre>
<p>Would that help in any kind?</p>

saba: <p>Hi Ptrblck,<br>
I need to apply histogram I used</p>
<pre><code class="lang-auto">
            xxx=torch.histc(CMBMASKGaussy,100, min=0, max=1, out=None)
</code></pre>
<p>But I need to know the bins elements too. How I can extract the bins elements from this function exactly as it is used?</p>
<p>As an another way I used</p>
<pre><code class="lang-auto">FFBins=[]
for jj1 in np.arange(0,1.03,.012):
     FFBins.append(jj1)

     hist_r = np.histogram(CMBMASKGaussy.squeeze(1).view(-1).detach().numpy(),bins=FFBins,range=[0 ,1])

            count_r = hist_r[0]         

            bins = hist_r[1]
</code></pre>
<p>why the number of the bins is one more than the count_r ? They should be same.I need to be precise, would you please help me with that which of them god to use?</p>