akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
I don't find where the mistake is.Who can help me,please

haoxiansheng: <p>import torch.nn as nn</p>
<p>N, D_in, H, D_out = 64,1000,100,10</p>
<p><span class="hashtag">#随机创建一些训练数据</span><br>
x = torch.randn(N, D_in)<br>
y = torch.randn(N, D_out)</p>
<p>class TwoLayerNet(torch.nn.Module):<br>
def <strong>init</strong>(self, D_in, H, D_out):<br>
super(TwoLayerNet, self).<strong>init</strong>()<br>
self.Linear1 = torch.nn.Linear(D_in, H, bias=False)<br>
self.Linear2 = torch.nn.Linear(H, D_out, bias=False)</p>
<pre><code class="lang-auto">def forward(self, x):
    y_pred = self.linear2(self.linear1(x).clamp(min=0))
    return y_pred
</code></pre>
<p>model = TwoLayerNet(D_in, H, D_out)<br>
loss_fn = nn.MSELoss(reduction=‘sum’)<br>
learning_rate = 1e-4<br>
optimizer = torch.optim.Adam(model.parameters(),lr=learning_rate)</p>
<p>for it in range(500):</p>
<h1>Forward pass</h1>
<p>y_pred = model(x)</p>
<pre><code class="lang-auto"># compute loss
loss = loss_fn(y_pred, y)  # computation graph
print(it, loss.item())

optimizer.zero_grad()
# Backward pass
loss.backward()

# update model parameters
optimizer.step()
</code></pre>
<p>ModuleAttributeError Traceback (most recent call last)<br>
in<br>
24 for it in range(500):<br>
25 # Forward pass<br>
—&gt; 26 y_pred = model(x)<br>
27<br>
28 # compute loss</p>
<p>E:\anaconda3\envs\sbw\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)<br>
725 result = self._slow_forward(*input, **kwargs)<br>
726 else:<br>
→ 727 result = self.forward(*input, **kwargs)<br>
728 for hook in itertools.chain(<br>
729 _global_forward_hooks.values(),</p>
<p>in forward(self, x)<br>
14<br>
15 def forward(self, x):<br>
—&gt; 16 y_pred = self.linear2(self.linear1(x).clamp(min=0))<br>
17 return y_pred<br>
18</p>
<p>E:\anaconda3\envs\sbw\lib\site-packages\torch\nn\modules\module.py in <strong>getattr</strong>(self, name)<br>
776 if name in modules:<br>
777 return modules[name]<br>
→ 778 raise ModuleAttributeError("’{}’ object has no attribute ‘{}’".format(<br>
779 type(self).<strong>name</strong>, name))<br>
780</p>
<p>ModuleAttributeError: ‘TwoLayerNet’ object has no attribute ‘linear2’</p>

Monne: <p>Maybe because on forward you called linear with lowercase but you define it with first letter uppercase</p>

haoxiansheng: <p>Yes,you are right.Thank you very much.Thank you</p>
ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all()

wsy: <p>data=pd.read_csv(‘附件1：sale_info.csv’,parse_dates=[‘date_rcd’])<br>
data[‘year’]=data.date_rcd.dt.year<br>
data[‘month’]=data.date_rcd.dt.month<br>
data[‘day’]=data.date_rcd.dt.day<br>
data=pd.DataFrame(data)<br>
data=data[data[‘year’]==2018]<br>
data=data[((data[‘month’] &lt;10.0)&amp;(data[‘month’] &gt;= 7.0))or((data[‘month’]==10.0)&amp;(data[‘day’]==1.0))]<br>
data.eval(‘profit=s*real_cost’,inplace=True)<br>
data=data.groupby(‘skc’)[‘profit’].sum()<br>
data=data.sort_values(ascending=False)<br>
print(data)<br>
data.to_csv(“Test_1.csv”, encoding=“utf-8-sig”, header=True, index=True)</p>

Kushaj: <p>Use <code>|</code> for or and <code>&amp;</code> for and.</p>

haviernick: <p><a href="http://net-informations.com/ds/pd/default.htm" rel="noopener nofollow ugc">Pandas</a> follows the numpy convention of raising an error when you try to convert something to a bool. This happens in a if or when using the boolean operations, and, or, or not. It is not clear what the result of.</p>
<p>example</p>
<p>5 == pd.Series([12,2,5,10])</p>
<p>The result you get is a Series of booleans, equal in size to the pd.Series in the right hand side of the expression. So, you get an error. The problem here is that you are comparing a pd.Series with a value, so you’ll have multiple True and multiple False values, as in the case above. This of course is ambiguous, since the condition is neither True or False. You need to further aggregate the result so that a single boolean value results from the operation. For that you’ll have to use either any or all depending on whether you want at least one (any) or all values to satisfy the condition.</p>
<pre><code class="lang-auto">(5 == pd.Series([12,2,5,10])).all()
# False
</code></pre>
<p>or</p>
<pre><code class="lang-auto">(5 == pd.Series([12,2,5,10])).any()
# True
</code></pre>
What is the best way to pass in a tensor that could optionally be None to a function?

Carol_Ye_Liu: <p>I defined a forward function in a module. The forward function checks if the noise input parameter is None or not, if None then create its own noise; if not None, then just use the input noise. However, this could not be achieved using the following code, c10::optional&lt;<a>at::Tensor</a>&gt; could not be converted to torch::Tensor through " mynoise = noise;". What is the best way to pass in a tensor that could optionally be None to a function? Thanks!</p>
<p>torch::Tensor forward(torch::Tensor  image, c10::optional &lt;<a>at::Tensor</a>&gt; noise){<br>
torch::Tensor mynoise;<br>
if(noise == None){<br>
auto batch = image.size(0);// <em>, height, width = image.shape<br>
auto height = image.size(2);<br>
auto width = image.size(3);<br>
mynoise = image.new_empty({batch, 1, height, width}).normal</em>();<br>
}else{<br>
mynoise = noise;<br>
}<br>
return image + weight *  mynoise;<br>
}</p>

Carol_Ye_Liu: <p>Does anyone have any suggestions?</p>

tom: <p>The typical thing is to just take a tensor argument (<code>const Tensor&amp;</code> or <code>Tensor</code>). Then test <code>t.defined()</code>.<br>
None is translated to an undefined Tensor.</p>
<p>Best regards</p>
<p>Thomas</p>

Carol_Ye_Liu: <p><a class="mention" href="/u/tom">@tom</a> Many thanks! To clarify, in my function forward, i need to check for (noise.defined()) instead of (noise==None) . But if I want to call my function “forward” with noise undefined, how do i do that without setting a default value for noise in the function? I just do " torch::Tensor noise_dummy;" and pass in noise_dummy?</p>

tom: <p>That would work, but the most succinct way of passing “None” would probably be using the empty init list <code>{}</code> .</p>
<p>Best regards</p>
<p>Thomas</p>

7starsea: <p>I am using pytorch 1.6.0 and my cpp code is like</p>
<pre><code class="lang-auto">void fit(const at::Tensor &amp; x, const at::Tensor &amp; y, const at::Tensor &amp; w){
   if(w.defined()){
   
   }else{
  
   }
}
</code></pre>
<p>and the python binding code (using pybind11) is like</p>
<pre><code class="lang-auto">m.def("fit", &amp;fit, py::arg("x"), py::arg("y"), py::arg("w")=py::none())
</code></pre>
<p>The compiling is fine. However,  I failed to call it in python using <code>fit(x, y) </code> or <code>fit(x,y, None)</code>.</p>
<p>The error is<br>
<code>TypeError: fit(): incompatible function arguments.</code></p>
<p>thanks.</p>

tom: <p>What this is about is that None in Python maps to an undefined <code>torch::Tensor</code> in C++. The the <code>m.def</code> is on the C++ side already,  so the default argument should be the default constructed <code>torch::Tensor()</code>.</p>

7starsea: <p>I tried <code>py::arg("w")=at::Tensor()</code>, it still fails and the error is the same.</p>
MobileNetV2 converted with PTQ is very poor accuracy

crook52: <p>Hi,<br>
I followed <a href="https://pytorch.org/tutorials/recipes/quantization.html?highlight=quantization" rel="noopener nofollow ugc">tutorials/quantization </a>and tried to PTQ MobileNetV2 from torchvision.<br>
I have quantized with the following script.<br>
However, the accuracy is very poor.<br>
What can I do to improve the accuracy?</p>
<ul>
<li>original model<br>
Top1: 64.1% Top5:85.8%</li>
<li>quantized model<br>
Top1: 22.0% Top5:41.9%<br>
tested by ImageNet-1K</li>
</ul>
<pre><code class="lang-auto">model = torchvision.models.quantization.mobilenet_v2(pretrained=True)

def calibrate_model(model, loader, device=torch.device("cpu:0")):
    model.to(device)
    model.eval()
    if len(loader) == 2:
        print("data size is ", len(loader[0]))
        inputs = loader[0].to(device)
        labels = loader[1].to(device)
    else:
        for inputs, labels in tqdm(loader):
            inputs = inputs.to(device)
            labels = labels.to(device)
    _ = model(inputs)

model.eval()
backend = "fbgemm"
torch.backends.quantized.engine = backend
model.fuse_model()
model.qconfig = torch.quantization.QConfig(
    activation=torch.quantization.MinMaxObserver.with_args(reduce_range=True), #default_observer
    weight=torch.quantization.PerChannelMinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric) #default_per_channel_weight_observer
)
torch.quantization.prepare(model, inplace=True)
calibrate_model(model=model, loader=valid_queue)
torch.quantization.convert(model, inplace=True)

</code></pre>
<p>Thank you.</p>

Zafar: <p>You can try QAT or dynamic quantization. I think QAT might get your accuracy slightly higher. In addition to that, if you leave the qconfig as default, it might give you slightly better numerics.</p>
Convert Pytorch Model to Publishable Figure

aksg87: <p>How do most people create figures for their models? I can visualize my model structure in Tensorboard and would now love to export it into a clean SVG or pdf.</p>
<p>Few screenshots of the encoder and decoder taken from Tensorboard.</p>
<p>encoder:<br>
                    <a href="https://dl.dropboxusercontent.com/s/jtoqfvatztahsp6/Screen%20Shot%202021-03-14%20at%206.05.38%20PM.png?dl=0" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://dl.dropboxusercontent.com/s/jtoqfvatztahsp6/Screen%20Shot%202021-03-14%20at%206.05.38%20PM.png?dl=0" width="371" height="500">
          </a>

</p>
<p>decoder:<br>
                    <a href="https://dl.dropboxusercontent.com/s/oa87dlujbb83t1c/Screen%20Shot%202021-03-14%20at%206.06.38%20PM.png?dl=0" target="_blank" rel="noopener nofollow ugc" class="onebox">
            <img src="https://dl.dropboxusercontent.com/s/oa87dlujbb83t1c/Screen%20Shot%202021-03-14%20at%206.06.38%20PM.png?dl=0" width="690" height="435">
          </a>

</p>

tom: <aside class="quote no-group" data-username="aksg87" data-post="1" data-topic="114785">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aksg87/48/15602_2.png" class="avatar"> aksg87:</div>
<blockquote>
<p>How do most people create figures for their models?</p>
</blockquote>
</aside>
<p>I can’t speak for most people but I have an <a href="https://github.com/t-vi/pytorch-tvmisc/blob/master/hacks/visualize-jit-models.ipynb" rel="noopener nofollow ugc">ad-hoc tool for creating dot</a> that I use with JITed models.<br>
I found that in the end, I always want to fiddle with the detail level so I never made it a generic thing. On the other hand making it [interactive (double click to expand and reduce) <a href="https://static.laszlokorte.de/graph/svg.html" rel="noopener nofollow ugc">seems possible but quite a bit of work</a>.</p>
<p>Best regards</p>
<p>Thomas</p>
How can I remove data from Dataset

lxy1997: <pre><code class="lang-auto">index = [1,2,3,....]
train_tensor.shape = torch.size(128,30,100)
train_label.shape=torch.size(128,1,1) 
train_set = dataset(train_tensor,train_label) 
</code></pre>
<p>Now , I want to remove data form Train_set using index?<br>
I had tried this function:</p>
<pre><code class="lang-auto">def tensor_remove(tensors,index):
    # print("tensors:",tensors.shape[0])
    sub_tensors = torch.empty([0,30,100])
    if index == []:
        return tensors
    if tensors.shape[1] == 1:
        sub_tensors = torch.empty([0,1,1])
    last_idx = -1
    for idx in index:
        if idx != last_idx+1:
            sub_tensors = torch.cat((sub_tensors,tensors[last_idx+1:idx]),0)
        if idx == index[-1]:
            sub_tensors = torch.cat((sub_tensors,tensors[idx+1:]),0)
        last_idx = idx
    # print("sub_tensor:",sub_tensors.shape[0])
    return sub_tensors
</code></pre>
<p>It words!But when I running train.py ,it rasing CUDA out of memerys! so ,I guess maybe the function caused that problem! Please help me! thanks!</p>
grad_fn=<DivBackward0>

sugh7020: <p>What does grad_fn = DivBackward0 represent?<br>
I have two losses:</p>
<pre><code class="lang-auto">L_c -&gt; tensor(0.2337, device='cuda:0', dtype=torch.float64)

L_d -&gt; tensor(1.8348, device='cuda:0', grad_fn=&lt;DivBackward0&gt;)
</code></pre>
<p>I want to combine them as:</p>
<pre><code class="lang-auto">L = L_d + 0.5 * L_c
optimizer.zero_grad()
L.backward()
optimizer.step()
</code></pre>
<p>Does the fact that one has <em>DivBackward0</em> and other doesn’t  cause an issue in the backprop?</p>

InnovArul: <aside class="quote no-group" data-username="sugh7020" data-post="1" data-topic="114794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/7feea3/48.png" class="avatar"> sugh7020:</div>
<blockquote>
<p><code>L_c -&gt; tensor(0.2337, device='cuda:0', dtype=torch.float64)</code></p>
</blockquote>
</aside>
<p><code>L_c</code> doesn’t have a connected computation graph, as there is no gradient function present.<br>
i.e., <code>optimizer.step()</code> would try to minimize only <code>L_d</code>.</p>
Debugging memory allocations in `torch.autograd.grad`

Linux-cpp-lisp: <p>Hi all,</p>
<p>We have a model that involves an autograd call as part of its forward pass, something along the lines of:</p>
<pre><code class="lang-python">return torch.autograd.grad(
            [prev.sum()],
            wrt_tensor,
            create_graph=True,  # needed to allow gradients of this output
        )[0]
</code></pre>
<p>When we scale the hidden dimension of our model we have started to get CUDA out-of-memory errors with enormous allocations at this call:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 20.54 GiB (GPU 0; 11.17 GiB total capacity; 831.28 MiB already allocated;
 9.91 GiB free; 886.00 MiB reserved in total by PyTorch)
</code></pre>
<p>This happens the very first time this call is ever made, and does not make much sense: if the whole model, data, compute graph, etc. fit in 886 MB, why does autograd need to allocate 20GB?</p>
<p>Is there a way to determine which part of the compute graph is causing <code>autograd</code> to make this allocation? The memory profiler unhelpfully suggests that <code>aten::empty</code> is responsible, and the python debugger can’t go into autograd itself.</p>
<p>Thank you for your help!</p>

albanD: <p>Hi,</p>
<p>You can enable anomaly mode. That will show you the forward op that corresponds to the one that is failing in the backward.<br>
Can you share this trace?</p>

Linux-cpp-lisp: <p>Hi <a class="mention" href="/u/alband">@albanD</a>,</p>
<p>That ended up letting me find the issue — initially it hadn’t seemed useful because the offending code was automatically generated and wasn’t available for the traceback. Temporarily changing that made it possible to see which line was responsible and resolve the issue.</p>
<p>(For us, it was an issue with how we had expressed a broadcasting operation.)</p>
<p>Thanks!</p>
Interpolation padding

adizhol: <p>What kind of padding does pytorch use for torch.nn.functional.interpolation, and how to control it?<br>
For example, how to set reflectace padding or zero-padding?</p>

ptrblck: <p>I don’t think that <code>F.interpolate</code> uses any padding, as it should create the indices corresponding to the desired output shape inside the valid input size.<br>
If you want to pad the input manually, you could use <code>F.pad</code>, where <code>mode</code> indicates the desired padding mode.</p>
Why would one shuffle the test data?

TinfoilHat0: <p>I’m aware uniformly shuffling training data results in good batches, and hence helps training.</p>
<p>However, is there really a point in shuffling test data?</p>

RoySadaka: <p>Shuffling data <strong>prior</strong> to Train/Val/Test splitting serves the purpose of reducing variance between train and test set.</p>
<p>Other then that, there is no point (that I’m aware of) to shuffle the test set, since the weights are not being updated between the batches.</p>
<p>Do you have a specific use case when you encountered shuffled test data?</p>
<p>Roy</p>

aguennecjacq: <p>Your test/validation data is meant to measure how good your model is. Hence, choosing to shuffle or no not your test/validation data set depends on what you are choosing to measure.<br>
For example, if you are extracting a information that is batch dependent (e.g mean loss on the batch) or you have a component in your neural network that is batch dependent, then you should shuffle for consistency.</p>
Gradcheck: "object has no attribute 'is_sparse'" in

Thomas_Ahle: <p>I’m trying to run the <code>MulConstant</code> code from <a href="https://pytorch.org/docs/master/notes/extending.html" rel="noopener nofollow ugc">Extending Pytorch</a></p>
<pre><code class="lang-auto">class MulConstant(Function):
    @staticmethod
    def forward(ctx, tensor, constant):
        # ctx is a context object that can be used to stash information
        # for backward computation
        ctx.constant = constant
        return tensor * constant

    @staticmethod
    def backward(ctx, grad_output):
        # We return as many input gradients as there were arguments.
        # Gradients of non-Tensor arguments to forward must be None.
        return grad_output * ctx.constant, None
</code></pre>
<p>In particular, I’m testing the gradcheck function</p>
<pre><code class="lang-auto">input = torch.randn(20, requires_grad=True)
print(gradcheck(MulConstant, (input, 2), eps=1e-6, atol=1e-4))
</code></pre>
<p>However, I get the error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
.../site-packages/torch/autograd/gradcheck.py", line 136, in get_analytical_jacobian
    if output.is_sparse:
AttributeError: 'MulConstant' object has no attribute 'is_sparse'
</code></pre>
<p>I can’t figure out what’s going on. I can’t find any documentation on this <code>is_sparse</code> attribute. The extending pytorch documentation doesn’t mention it either.</p>
<p>(I’m on torch 1.8.0)</p>

Thomas_Ahle: <p>Never mind. Apparently, <code>gradcheck</code> needs to be called with <code>MulConstant.apply</code>.</p>
Scale each instance by mask area in L1 loss

ZimoNitrome: <p>L1 loss by default uses <code>reduction=mean</code> and weighs loss on differently sized instances.</p>
<p>But consider the following scenario:<br>
I have a set size for batches and image dimensions, but each image has different masks where only a certain region is of interest.<br>
I think it could be useful to scale each instance’s loss by the area of the region of interest (i.e. number of ones in the mask).</p>
<p>I don’t believe elements which are exactly zero are ignored in L1Loss, are they?</p>
<p>Could something like this work?</p>
<pre><code class="lang-python"># Assume 'input', 'output', 'mask' all are NCHW

output = model(input)
diff = input*mask - output*mask
weighted_diff = mask.sum([1,2,3]) * diff
loss = L1_loss(weighted_diff, torch.zeros_like(weighted_diff))
</code></pre>

InnovArul: <p>I am not sure about the use case.<br>
But do you think, the losses from the images with “less mask area” should be weighed higher? If not, the model might get away by ignoring them.<br>
i.e.,</p>
<pre><code class="lang-auto">output = model(input)
abs_diff = torch.abs((input - output)*mask).sum([1,2,3])
loss = abs_diff / (mask.sum([1,2,3]) + eps)  # eps ~ 10-6
</code></pre>

ZimoNitrome: <p>Ah yes you are correct!<br>
I meant scaling it by the inverse scale of the mask which is what I assume you are referring to in the last line of code.</p>
<p>I am curious if there was some torch component or loss function argument that did this “behind the scenes”. I think it might just be better to do it explicitly like you suggested.</p>

InnovArul: <p>I haven’t come across a specific implementation in torch that gives this functionality so far.<br>
You might need to look around in the docs.</p>
Image Data Pre-processing

Sahil_malik: <p>Hi,</p>
<p>I am dividing my pixels training and test data by 255.0 in preprocessing step. I am bit confused about min max normalization. Is it same thing but different name which actually i am doing?</p>
<p>Thank you!</p>

InnovArul: <p>AFAIK, min-max normalization is to rescale the image pixel values to be between [0, 1].<br>
The formula is,</p>
<pre><code class="lang-auto">scaled_img = (img - img.min()) / (img.max() - img.min())
</code></pre>
<p>In case of images ranging [0, 255]. <code>img.min() = 0, img.max() = 255</code>.<br>
Thus, min-max normalization for these images is to divide by 255. (this assumes that image range is [0, 255])</p>
<p>Hope this helps!</p>

Sahil_malik: <p>Thank you…</p>
<p>So you mean diving pixel data by 255.0 is min max normalization. My image data values range from 0 to 255, that’s why I am dividing with 255.0</p>

InnovArul: <p>You are right. Sometimes, the images might not exactly range from [0, 255]. i.e, it may be [10, 250].<br>
If you want to min-max normalize according to the definition, you have to apply the formula.</p>
Parameter to save checkpoint

Subhankar_Ghosh: <p>I am trying to perform:</p>
<pre><code class="lang-auto">args = ParseCmdLineArguments()
# A few lines of code
path = os.path.join(args.save, 'trained_model-%05d.ckpt' % epoch)
torch.save(depth_net.state_dict(), path)
</code></pre>
<p>where my parameters passed to edit configuration in PyCharm are:</p>
<blockquote>
<p>–batch_size 1 --dataset_pickle_file some.pkl --learning_rate 1e-4 --save ‘./’</p>
</blockquote>
<p>I am getting this error:</p>
<pre><code class="lang-auto">unrecognized arguments: --save './'
</code></pre>
<p>Can you all please suggest how I can add the save attribute to args to save the checkpoint in the same folder? I am passing the arguments via edit configuration on PyCharm.<br>
Thanks!</p>

Dwight_Foster: <p>Try removing the single apostrophes around the ./ and setting the default type to a string when you define the args. Like this:</p>
<pre><code class="lang-auto">parser.add_argument("--save", type=str)
</code></pre>
<p>then</p>
<pre><code class="lang-auto">–batch_size 1 --dataset_pickle_file some.pkl --learning_rate 1e-4 --save ./
</code></pre>

Subhankar_Ghosh: <p>Hi <a class="mention" href="/u/dwight_foster">@Dwight_Foster</a> , Thanks for helping.</p>
<p>Can you please suggest where I can add:</p>
<pre><code class="lang-auto">parser.add_argument("--save", type=str)
</code></pre>
<p>into the IDE. I am running PyCharm. If you could suggest how you do it in your IDE I will try to find out how to do the same in PyCharm.</p>
<p>Thanks again!</p>

Dwight_Foster: <p>I run pycharm too. I would suggest instead of using the ParseCmdLineArguments() you use argparse. It will allow you to set a type default and many other things. You can find a tutorial <a href="https://docs.python.org/3/howto/argparse.html" rel="noopener nofollow ugc">here</a>, but for what you want to do you can just do this:</p>
<pre><code class="lang-auto">import argparse
parser = argparse.ArgumentParser()
parser.add_argument("--dataset_pickle_file", type=str)
parser.add_argument("--learning_rate", type=float)
parser.add_argument("--save", type=str)
args = parser.parse_args()
</code></pre>
What are good practices for using GANs with labels?

ZimoNitrome: <p>Creating a GAN I have read that using class labels can be extremely useful.<br>
I know that there are lots of different GAN architectures that utilize labels and I wonder if any of them have more proof to back up their strength than any other? And where to start looking for fair comparisons.</p>
<p>I know there are architectures like CGAN, InfoGAN, ACGAN.<br>
Some of these use labels as inputs to the discriminator. For low res images like MNIST it makes sense to concat a flattened image with the label but how does this translate to using larger images?</p>
<p>My current thought is to use multiple 2D convolutions, eventually feeding to linear layers that also takes the label inputs (OHE, binary input per neuron). Does this design make sense?</p>
RNN many to one

WaleedEsmail: <p>Dear PyTorch experts,</p>
<p>I am trying to understand the RNN and how to implement it as a classifier (Many to one). I’ve read many tutorials but still confused. One of these tutorials suggest to use the following:</p>
<pre><code class="lang-auto"># Recurrent neural network (many-to-one)
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_classes)
    
    def forward(self, x):
        # Set initial hidden and cell states 
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device) 
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)
        
        # Forward propagate LSTM
        out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size)
        
        # Decode the hidden state of the last time step
        out = self.fc(out[:, -1, :])
        return out
</code></pre>
<p>What I am confused about is why to use the last time step as an input for the dense layer. Also for a binary classification problem for example a <code>sigmoid</code> activation is applied on the output, why is it not applied here?! ( also for all tutorials that I read).</p>
<p>Thank you very much for your help</p>
<p>** tutorial from <a href="https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/recurrent_neural_network/main.py" rel="nofollow noopener">here</a></p>

Mahbub_Zaman: <p>In the batch first mode (which is what you have), the output has this meaning:<br>
batch * seq_len * vector size<br>
So with a -1 in the middle dimension, you are extracting the very last output of time series output sequence of the RNN. This final output represents the summary of the entire sequence and that is what you want for a many to one scenario. The other :'s just means you want that last output for all batch and all input vectors.</p>
How many threads will pytorch start for a convolution layer?

jqliu: <p>Hi, I’d like to know how many threads will pytorch start for a convolution layer or a convolution filter?<br>
thanks!</p>

tom: <p>Are you looking for <a href="https://pytorch.org/docs/stable/torch.html#parallelism" rel="noopener nofollow ugc">the parallelism functions</a>? They have C++ equivalents in ATen (ATen/Parallel.h).</p>
<p>Best regards</p>
<p>Thomas</p>

jqliu: <p>Thanks for your reply. the get_num_threads() function  returns the number of threads used for parallelizing CPU operations. However, I want to know how many threads are used for a convolution layer in GPU. Is there any method?<br>
Best regards</p>

tom: <p>No, on the GPU it would depend on many factors, the implementation (in particular the backend), the hardware specifics etc. Also it’s not quite clear what you want to have, do you want the number threads in a block or the number of threads times the number of blocks started in parallel?<br>
Threads per block and number of blocks are (save CuDNN benchmark=True and similar things) roughly deterministic in the sense that you get the same number of threads per block and blocks if you run them several times with the same shape and stride tensors on the same device, so you could try to fire up the profiling tool of your choice (e.g. nvprof) and let it dump the kernel invocations with the number of threads.</p>
<p>Best regards</p>
<p>Thomas</p>
Same Device error: Expected all tensors to be on the c, but found at least two devices, cuda:0 and cpu!

aswamy: <p>Getting device errors at line <code>all_rays_cos += cos(input_rays, target_ray)</code>  in the below function.<br>
I checked both tensors <code>input_rays</code> and <code>target_ray</code>, both are on <code>cuda:0</code>. I do not know which tensor is using CPU. Any help is very much useful? Thanks!</p>
<pre><code class="lang-auto">def compute_feat_weights(src_rays, all_rays, device=None):
    assert src_rays.shape[0] == all_rays.shape[0]  
    assert src_rays.shape[-1] == all_rays.shape[-1]
    SB = src_rays.shape[0]
    NS = src_rays.shape[1]
    H, W = src_rays.shape[2:4]
    target_ray_batch_size = all_rays.shape[1]
    input_rays = src_rays.view(SB, NS, -1, 8)
    cos = nn.CosineSimilarity(dim=3, eps=1e-6).to(device)  # similarity func
    all_rays_cos = torch.zeros(SB, NS, H*W)
    for idx in range(0, target_ray_batch_size):
        target_ray = all_rays[:,idx].unsqueeze(0).unsqueeze(1)
        print(input_rays.device, target_ray.device)
        **all_rays_cos += cos(input_rays, target_ray)** 
    feat_weights = torch.sum(all_rays_cos, dim=2)
    norm_feat_werigts = (feat_weights - torch.mean(feat_weights)) / torch.std(feat_weights)
    
    return norm_feat_werigts.view(SB, NS, 1)`Preformatted text`
</code></pre>

Dwight_Foster: <p>The problem is all_rays_cos is not on the gpu. When you define it here:</p>
<pre><code class="lang-auto">    all_rays_cos = torch.zeros(SB, NS, H*W)
</code></pre>
<p>you do not convert it to cuda. Just change that to this</p>
<pre><code class="lang-auto">    all_rays_cos = torch.zeros(SB, NS, H*W).to(device)
</code></pre>

aswamy: <p>Oops! I missed that. Thanks  a lot.</p>
Looping over model parameters in reverse

Kamil_Adamczewski: <p>When we loop over the parameters we use,</p>
<pre><code> for layer_name, param in net.named_parameters()
</code></pre>
<p>and the parameters are listed in the order they appear in the architecture. Is it possible to loop over these parameters in reverse order?</p>

Dwight_Foster: <p>You can do this</p>
<pre><code class="lang-auto">for layer_name, param in reversed(list(net.named_parameters())):
  print(layer_name)
</code></pre>
Question about torch.nn.Upsample() function

YDDDDG: <p>I used <em>torch.nn.Upsample()</em> function to upsample my 5D data, thus I write code in this way :<br>
<code>self.upsample_layer = torch.nn.Upsample(scale_factor=upsample, mode='trilinear', align_corners=True)</code><br>
However, a warning appears:<br>
<strong>WARNING:root: mode trilinear is not implemented yet, take it a zero op</strong><br>
it seems that trilinear mode is not implemented, but it doesn’t make sense.<br>
So What cases this warning? What should I do to resolve it?<br>
Thanks~</p>

Nikronic: <p>Hi,</p>
<p><code>nn.Upsample</code> has been deprecated. Use <code>torch.nn.functional.interpolate</code> instead.<br>
<a href="https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/nn.functional.html#torch.nn.functional.interpolate</a></p>
<p>Bests</p>
How to use fix randomGenerator in multithread?

scirocc19900305: <p>my code like:</p>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include "torch/torch.h"
int main() {
    torch::Tensor mu=torch::zeros(1);
    torch::Tensor std=torch::ones(1);
    std::vector&lt;double&gt; vc;
    vc.reserve(8);
#pragma omp parallel for num_threads(8) default(shared)
    for(int i=0;i&lt;8;++i){
        int tdidx=omp_get_thread_num();
        torch::manual_seed(1);
        c10::optional&lt;at::Generator&gt; generator;
        generator-&gt;set_current_seed(tdidx);// leads to the error
        vc[tdidx]=at::normal(mu,std,generator).item&lt;float&gt;();
        std::cout&lt;&lt;at::normal(mu,std).item&lt;float&gt;() &lt;&lt;""&lt;&lt;tdidx&lt;&lt;std::endl;
    }
    std::cout&lt;&lt;"!!!!!!!!!!!!!" &lt;&lt;std::endl;
    std::cout&lt;&lt; vc[0]&lt;&lt;std::endl;
    std::cout&lt;&lt; vc[1]&lt;&lt;std::endl;
    std::cout&lt;&lt; vc[2]&lt;&lt;std::endl;
    return 0;
}
</code></pre>
<p>and I got：“Process finished with exit code 139 (interrupted by signal 11: SIGSEGV)”<br>
any idea？</p>
How can pytorch achieve this one? Anyone help?thanks

cccwz: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/d/3d7ecae69509d9c7b9498d88c4ddc2ab7394d3eb.png" alt="QQ截图20210314204146" data-base62-sha1="8M0McTJhZHiSQmamRVPfc326H0L" width="644" height="258"></p>
Pytorch Benchmark docs Issue

saurabheights: <p><a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html#comparing-benchmark-results" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/recipes/recipes/benchmark.html#comparing-benchmark-results</a></p>
<p>The above links shows comparison of time taken by two different implementation of torch.dot method mul/sum and bmm. The table shows bmm takes longer when num_threads &gt; 1, but after table, docs mention: “The results above indicate that the version which reduces to bmm is better for larger tensors <em>running on multiple threads,</em> while for smaller and/or single thread code, the other version is better.”</p>
<p>However, bmm mostly outperferms mul/sum when running on larger tensors <em>on a single thread</em>. Did I miss anything here?</p>
Mini-batch within train loop

arjung: <p>Long story short, I cannot modify the input batch size of 128 of my data loader. When I do:</p>
<pre><code class="lang-auto">for batch_idx, o_t in enumerate(train_loader):
    o_t = o_t.to(device)
    y = model(o_t)
</code></pre>
<p>I get a CUDA out of memory error.</p>
<p>To get around this, I tried the following:</p>
<pre><code class="lang-auto">for batch_idx, o_t in enumerate(train_loader):
    mini_batch_size = 16
    y = []
    for mini_batch_idx in range(int(128/mini_batch_size)):
        start, end = mini_batch_idx*mini_batch_size, (mini_batch_idx+1)*mini_batch_size
        o_t_mini = o_t[start:end]
        o_t_mini = o_t_mini.to(device)

        y_mini = model(o_t_mini)

        o_t_mini, y_mini = o_t_mini.cpu(), y_mini.cpu()

        y.append(y_mini)

y = torch.cat(y, dim=0)
</code></pre>
<p>However, this does not help either, and I observe that GPU memory usage increases linearly after each forward pass, and not after the tensors are moved to the device. Moving the tensors back to cpu has no effect on GPU memory, either.</p>
<p>Any ideas why this is the case, and how I could get around this? Thanks!</p>

Valerio_Biscione: <p>Is this a very massive model? You may want to try with a simple toy module such as <code>nn.Conv2d</code> and check if it still crashes. Most of the GPU memory may be taken by the model, not the data</p>

arjung: <p>I see. But I don’t understand why it would increase with every forward pass – shouldn’t it remain constant once the model parameters have been initialized?</p>

Valerio_Biscione: <p>Unfortunately, I don’t remember why, but I also found in the past that the memory would increase for the first couple of epochs before stabilizing.</p>

ptrblck: <p>The increase in memory is expected, since you are storing each <code>y_mini</code> output in <code>y</code> alone with the complete computation graphs.<br>
If you want to reduce the memory usage, you could e.g. calculate the loss and gradients using <code>y_mini</code>.<br>
This would free the intermediate tensors (needed to calculate the gradients), as they are currently all stored on the GPU.</p>
Secondary GPU for display output

nicofish: <p>Hello I have been thinking about getting a secondary GPU for display output something like a GT710 which can be had new for around 70 dollars. The reason for this is that my computer lags terribly while I am training and becomes basically unusable due to high GPU usage.</p>
<p>I currently use a GTX1070 to do my training. I was wondering if using this secondary GPU would cause any issues when I run my training on my 1070?</p>
<p>Does anyone have any recommendations for a reasonable priced secondary GPU to use for display out?</p>
<p>thank you for your help</p>

ptrblck: <p>No, the small “display” GPU shouldn’t cause any issues and I’ve had a similar setup in the past in order to free the memory usage of the main GPU.</p>
Pytorch - Functional Interpolate mixing channels

alexandrepires: <p>Hello everyone,</p>
<p>I have the following issue regarding the use of functional interpolate in pytorch(my version is 1.7.1, running on Windows):</p>
<p>I want to downsample an image, on a scale factor of 2.<br>
The tensor of the original has the shape: [1 x 3 x 128 x 256]</p>
<p>The result of the interpolate is the following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fcebe4a141c9d3566c46f932d146096fcfc2cae0.png" alt="downsampled_image" data-base62-sha1="A5rII8bep3M2R2NTSAcjfN26LJK" width="640" height="480"></p>
<p>The tensor of the downsampled image has expected shape: [1 x 3 x 64 x 128]<br>
But the result seems all “mixed” up. It seems as its trying to draw each of the channels individually in the image.</p>
<p>The code that does this is as it follows(even changing the align_corners parameters isnt doing much):</p>
<pre><code class="lang-auto">    def scaling_pyramid(self, img, num_scales):
        scaled_imgs = []
        s = img.size()
        h = s[2]
        w = s[3]
        for i in range(num_scales):
            ratio = 2 ** (i + 1)
            nh = h // ratio
            nw = w // ratio
            print(img[0, :, :, :])
            scaled_imgs.append(torch.nn.functional.interpolate(img, size=[nh, nw], mode="bilinear", align_corners=False))
            print(scaled_imgs[i][0, :, :, :])
            current_tensor = scaled_imgs[i][0, :, :, :]
            print(current_tensor.shape)
            show_scaling_image = current_tensor.resize_(current_tensor.shape[1], current_tensor.shape[2], current_tensor.shape[0]).cpu().numpy()
            print(show_scaling_image.shape)
            plot.imshow(show_scaling_image)
            plot.show()
        return scaled_imgs
</code></pre>
<p>Any hints / help will be greatly appreaciated!</p>

alexandrepires: <p>Since im new, i cannot post 2 pieces of media on the post…</p>
<p>So, in this reply, I attach the original image.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/4/649dc746bfd21cba9a261aff7753491d72e7feb7.png" alt="original_image" data-base62-sha1="em5MIs5kZoi8RxOczi9KgmFCy6r" width="640" height="480"></p>

ptrblck: <p>I assume that the <code>resize_</code> operation fails in:</p>
<pre><code class="lang-python">current_tensor.resize_
</code></pre>
<p>and creates an interleaved tensor.<br>
Print the shape of <code>current_tensor</code> before applying this operation and make sure that you <code>permute</code> the dimensions properly before resizing the tensor.</p>
Need some help with my Double DQN implementation which plateaus long before reaching the Nature results

dominikschmidt: <p>I’m trying to replicate the Mnih et al. 2015/Double DQN results on Atari Breakout but the per-episode rewards (where one episode is a single Breakout game terminating after loss of a single life) plateau after about 3-6M frames with a per-ep reward of around 5-6:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65367f8834ce0749d447f247e383c3d1476ba07a.png" data-download-href="https://discuss.pytorch.org/uploads/default/65367f8834ce0749d447f247e383c3d1476ba07a" title="learning curves"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65367f8834ce0749d447f247e383c3d1476ba07a_2_690x181.png" alt="learning curves" data-base62-sha1="ermZ9nqhEV48rmS7KgbfZSpRyt4" width="690" height="181" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65367f8834ce0749d447f247e383c3d1476ba07a_2_690x181.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65367f8834ce0749d447f247e383c3d1476ba07a_2_1035x271.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65367f8834ce0749d447f247e383c3d1476ba07a_2_1380x362.png 2x" data-dominant-color="FEFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">learning curves</span><span class="informations">4266×1124 334 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It would be really awesome if anyone could take a quick look <a href="https://gist.github.com/schmidtdominik/3799130c7495b96f4cd9a1533fdd6083" rel="noopener nofollow ugc"><em>here</em></a> and check for any “obvious” problems. I tried to comment it fairly well and remove any irrelevant parts of code.</p>
<p>Things I have tried so far:</p>
<ul>
<li>DDQN instead of DQN</li>
<li>Adam instead of RMSProp (training with Adam doesn’t even reach episode reward &gt; 1, see gray line in plot above)</li>
<li>various learning rates</li>
<li>using exact hyperparams from the DQN, DDQN, Mnih et al 2015, 2013,… papers</li>
<li>fixing lots of bugs</li>
<li>training for more than 10M frames (most other implementations I have seen reach a reward about 10x mine after 10M frames)</li>
</ul>
<p>My goal ist to fully implement Rainbow-DQN but I would like to get DDQN to work properly first.</p>