akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Efficient implementation of Monte Carlo sampling in VAEs

Raphikowski: <p>Dear all,</p>
<p>I am implementing a VAE, where you approximate the expectation values by Monte Carlo sampling. In the original Kingma paper they just used one sample per batch but for me it turned out that I need several samples to get a  good estimate.<br>
The problem is just that drawing 10 Monte Carlo samples instead of just 1 increases the computation time per epoch by more than 100%. Im doing this sequentially in a for loop, so in each for loop iteration I draw a Monte Carlo sample then plug it into the loss function and calculate that.<br>
My guess is that somehow the algorithm also then backpropagates 10 times through the same networks otherwise I could explain why this increases the running time so much since drawing samples and calculating the loss isnt that big of a time factor.<br>
Do you have any ideas how this could be implemented more efficiently?</p>

alexandre-blanc: <p>I don’t know if you are still interested in this issue. You can draw a bunch of samples at once, storing them in an extra dimension. You can then flatten this dimension and make the batch go through the decoder. You then reshape the tensor to reinsert the extra dimension (it should be of size L, following the notations in Kingma &amp; Welling’s paper). You finally compute the likelihood and average across this dimension.</p>
<p>In principle, it sounds reasonable to expect that the running time goes up quite a bit. Imagine in each step the time it takes to perform the computations before the sampling is a, and the time after the sampling is b. For L = 1, the total time is a+b. If L&gt;1, then total time is a+L*b. If b &gt;&gt; a, then total time is roughly multiplied by L, which means 1000% for L = 10.</p>
<p>By batching the computation as I described, you are able to benefit from the parallel matrix multiplications performed by the GPU. So if your batch size is small enough, the running time should not change.</p>
<p>Keep in mind I am not an expert.</p>
<p>I have written an implementation of this algorithm, but in my case setting L to anything other than 1 completely breaks the algorithm. It might be a bug, or a feature of the problem I am trying to solve that I don’t understand yet.</p>
<p>Keep me posted.</p>
Weighted standard deviation of list of tensors

mustansarsaeed: <p>Hi, I have implementation of weighted <code>std</code> in <code>NumPy</code> as:</p>
<pre><code class="lang-auto">def weighted_std(average_of_tensor_list, list_of_tensors, list_num_samples):
     list_of_tensors_new = np.array([(list_of_tensors[i] - average_of_tensor_list)**2 for i in range(len(list_of_tensors))], dtype="object")
     variance = np.average(list_of_tensors_new, weights=list_num_samples, axis=0)
     std = np.array([np.sqrt(variance[i]) for i in range(len(variance))], dtype="object")
     return std
</code></pre>
<p>I am trying to convert this into equivalent <code>PyTorch</code> code, can anyone please guide, what should be the way?</p>
Applying mask using torch.where function

mustansarsaeed: <p>Hi,</p>
<p>I am trying to apply condition on tensors. What I am trying to achieve is: When mask is true then use the value from X otherwise Y. Logic works fine using <code>np.where</code> function. I am trying to achieve same using <code>torch.where</code>. Anything i am missing on the following?</p>
<p><code>weights_x</code> and <code>weights_y</code> both have size:</p>
<pre><code class="lang-auto">torch.Size([32, 1, 5, 5])
torch.Size([32, 1, 5, 5])
</code></pre>
<p><code>mask= th.tensor([False,  True, False,  True,  True, False, False,  True,  True,  True,         False,  True, False,  True,  True,  True, False,  True,  True, False,         False,  True, False, False, False, False,  True, False, False, False,          True,  True])</code></p>
<p><strong>Numpy version</strong><br>
<code>np.where(mask, weights_x, weights_y)</code></p>
<p><strong>Pytorch version</strong><br>
<code>th.where(mask == True, weights_x, weights_y)</code></p>
<p><strong>Error:</strong></p>
<pre><code class="lang-auto">if isinstance(args_, tuple):
--&gt; 414             response = command_method(*args_, **kwargs_)
    415         else:
    416             response = command_method(args_, **kwargs_)

RuntimeError: The size of tensor a (32) must match the size of tensor b (5) at non-singleton dimension 3
</code></pre>

ptrblck: <p><code>numpy</code> as well as PyTorch will both fail with a shape mismatch error so you would need to unsqueeze the <code>mask</code> tensor:</p>
<pre><code class="lang-python">weights_x = torch.zeros([32, 1, 5, 5])
weights_y = torch.ones([32, 1, 5, 5])
mask= torch.tensor([False, True, False, True, True, False, False, True, True, True, False, True, False, True, True, True, False, True, True, False, False, True, False, False, False, False, True, False, False, False, True, True])

np.where(mask, weights_x, weights_y) # fails
&gt; ValueError: operands could not be broadcast together with shapes (32,) (32,1,5,5) (32,1,5,5) 

torch.where(mask == True, weights_x, weights_y) # fails
&gt; RuntimeError: The size of tensor a (32) must match the size of tensor b (5) at non-singleton dimension 3

# works
arr = np.where(mask[:, None, None, None], weights_x, weights_y)
out = torch.where(mask[:, None, None, None], weights_x, weights_y)

print((arr == out.numpy()).all())
&gt; True
</code></pre>

mustansarsaeed: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the reply. It worked fine. How can I apply this <code>where</code> condition along <code>y-axis</code>? I have mask of shape <code>torch.Size([2048])</code> and tensor of shape <code>torch.Size([62, 2048])</code>. I want to apply the mask along y-axis, can you please comment? Thanks.</p>
Regression - pass vector & numeric feature as input

Shilpa: <p>I need help in a regression scenario.I have 12 input features. 4 of which are coordinates(each is a vector) in XYZ plane (x1,y,z1), (x2,y2,z2),(x3,y3,z3), (x4,y4,z4) and remaining 8 are numeric features. I have 2 numeric target variables.</p>
<p>The coordinates refer to 4 object coordinates in space. These are not images, just coordinates in space.</p>
<p>How can I input the 4 coordinates into the neural network along with 8 other numeric features?</p>
<p>I know how to pass just the 8 numeric features while training. But, I am not sure how to pass the 4 coordinates that are vectors.</p>
<p>Thanks.</p>
ValueError: Expected input batch_size (1) to match target batch_size (100)

AP_M: <p>I am trying to execute my dataset with batch size = 100. It works fine with batch size = 1 but with 100, it gives the error as: ValueError: Expected input batch_size (1) to match target batch_size (100).</p>
<pre><code class="lang-auto"> class Net(nn.Module):
        def __init__(self):
            super(Net, self).__init__()
            self.conv1 = nn.Conv1d(100, 1, kernel_size=1)
            self.dropout = nn.Dropout2d()
            self.fc1 = nn.Linear(32, 16)
            self.fc2 = nn.Linear(16, 1)
            self.hybrid = Hybrid(qiskit.Aer.get_backend('qasm_simulator'), 100, np.pi / 2)
    
        def forward(self, x):
            x = F.relu(self.conv1(x))
            x = self.dropout(x)
            x = x.view(x.size(0), -1)
            x = F.relu(self.fc1(x))
            x = self.fc2(x)
            x = self.hybrid(x)
            return torch.cat((x, 1 - x), -1)
    ```
    The sizes of data, label and output where output = model(data) are
    torch.Size([100, 32])
    torch.Size([100])
    torch.Size([1, 2])
    ```
    How to make the size of output same as data size?
</code></pre>
RuntimeError: Found dtype Long but expected Float for YOLO recreation

bindas1: <p>Hi,</p>
<p>I’m a 4th year CS student and as part of my engineering thesis I’m trying to recreate YOLO algorithm for object detection.</p>
<p>Despite making sure that both targets and inputs are converted to float and device before training I seem to experience this type of error and can’t solve it on my own.</p>
<p>The full source code is available in my public <a href="https://github.com/bindas1/YOLO-recreated/blob/master/Forward_propagation.ipynb" rel="noopener nofollow ugc">github repository</a></p>
<p>I believe that the problem may be relevant to the advanced loss function, but my debugging hasn’t helped.</p>
<p>Maybe someone could give me some tips as to how could I solve this problem?</p>

tom: <p>So error you hit is in the backward. The first thing I would recommend to do for this is use the (very slow) <code>ẁith torch.anomaly_detection()</code> mode as it will give you the backtrace of the backward.</p>
<p>Best regards</p>
<p>Thomas</p>

bindas1: <p>After debugging with <code>torch.anomaly_detection</code> I have added <code>.to(float)</code> in almost all tensors in loss and it works now!<br>
Thank You <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

tom: <p>Glad you solved it. If you stare down the backtrace (in the bottom half) to find the bit that isn’t Jupyter nor PyTorch internals but your code, the crucial information is the line with <code> no_object_loss = LAMBDA_NOOBJ * self.mse(</code> so likely it was <code>not_exists_object_filter</code> that was not float.</p>
<p>Unsolicited advice: If you use triple backticks (```) before and after the code in forum posts, it’ll use better formatting.</p>
Access the 32int MAC value

rawansn: <p>The pytorch quantization converts the INT32 MAC value into 8b in the backend. How can we access this compute layer INT32 value prior to conversion.</p>

jerryzh168: <p>unfortunately int32 value for fbgemm/qnnpack is not accessible from outside. Why do you need this?</p>

rawansn: <p>I need to impose some limits on the Accumulated values, how would that be feasible in the quantized model format?</p>

jerryzh168: <p>can you describe the whole flow? Are you trying to impose the limit on the kernel level or when people train the model?</p>
<p>If you need a kernel that imposes some limits on the int32 value, then I think the best thing to do is to reimplement the kernel (possibly by calling fbgemm implementations if you need high performance: <a href="https://github.com/pytorch/FBGEMM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/FBGEMM: FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/</a>)</p>

jerryzh168: <p>actually you might be able to modify the operator implementation a little bit and implement your own version of ops like quantized::conv: <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/quantized/cpu/qconv.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/qconv.cpp at master · pytorch/pytorch · GitHub</a></p>
Can we read logs to TensorBoard Using PyTorch

legendu: <p>I wrote logs to TensorBoard using PyTorch but want to generate customized visualization. I know TensorBoard provide links to download files, but I want know whether it is possible to directly read TensorBoard logs using PyTorch without installing the TensorBoard library.</p>

jerryzh168: <p>this is probably mislabeled, are you planning to post in <a href="https://discuss.pytorch.org/c/tensorboard/28" class="inline-onebox">tensorboard - PyTorch Forums</a></p>

jerryzh168: <p>will mark as solved since this is not related to quantization, please make a new post in tensorboard subcategory</p>
Where is function MulBackward0()'s definition?

zfzhang: <p>Hi,<br>
I often encounter messages like “RuntimeError: Function ‘MulBackward0’ returned nan values in its 0th output”.  I can make a wild guess that one of the multiplicand is too large.<br>
I want to find the exact definition of MulBackward0, but can’t seem to find it in Github repository.</p>
<p>Thanks very much!</p>
Class Activation Mapping(CAM) In Android

MoAmri: <p>I could draw the CAM using this tutorial<br>
<a href="http://snappishproductions.com/blog/2018/01/03/class-activation-mapping-in-pytorch.html" rel="noopener nofollow ugc">Class Activation Mapping In PyTorch · Ian Pointer (snappishproductions.com)</a><br>
, but I was wondering if I can draw CAM for predicted image in android.</p>
Cuda running out of memory when trying to fit two GPT2 Medium models in one GPU?

azlaanmsamad: <p>I need to train two GPT2-medium models simultaneously such that output of one needs to be used by other and so on recursively.<br>
I unfortunately cannot fit both models on a single GPU along with the gradient calculations during training.</p>
<p>I can think of two ways to solve this:</p>
<ol>
<li>Split the model itself into several smaller parts and put them into multiple GPUs and train.</li>
<li>Put the two GPT-2 on two different GPUs and train them.</li>
</ol>
<p>However, I do not know the feasibility of both the options and its affect on the performance of the model.<br>
Is it possible to implement both the options on pytorch? If not, can you suggest me other ways to do so?</p>
<p>Thank you in advance.</p>
Cannot input data to GPU

zlw: <p><span class="hashtag">#include</span> &lt;opencv2/opencv.hpp&gt;<br>
<span class="hashtag">#include</span> &lt;torch/script.h&gt;<br>
<span class="hashtag">#include</span> &lt;torch/torch.h&gt;<br>
<span class="hashtag">#include</span> <br>
<span class="hashtag">#include</span></p>
<p>//测试环境是否有问题<br>
int main()<br>
{<br>
std::cout &lt;&lt; “cuda::is_available():” &lt;&lt; torch::cuda::is_available() &lt;&lt; std::endl;// CUDA是否可用<br>
std::cout &lt;&lt; “cudnn_is_available():” &lt;&lt; torch::cuda::cudnn_is_available() &lt;&lt; std::endl;// cuDNN 是否可用<br>
std::cout &lt;&lt; “cuda::device_count():” &lt;&lt; torch::cuda::device_count() &lt;&lt; std::endl;// 可使用的GPU个数</p>
<pre><code>torch::DeviceType device_type = at::kCPU; // 定义设备类型
if (torch::cuda::is_available())
    device_type = at::kCUDA;
std::cout &lt;&lt; device_type &lt;&lt; std::endl;

//torch::Tensor tensor = torch::rand({ 5,3 });
torch::Tensor tensor = torch::rand({ 5,3 }).to(device_type);
std::cout &lt;&lt; tensor &lt;&lt; std::endl;

return 0;
</code></pre>
<p>}</p>

ptrblck: <p>Could you describe your issue a bit more? Are the <code>cout</code>s returning False?<br>
If so, did you install the <code>libtorch</code> package with CUDA support?</p>
Avoid writeable warning when copying numpy into pytorch tensor

KarelPeeters: <p>Hey,</p>
<p>I’m trying to optimize my data loading pipeline, and I’ve run into this warning:</p>
<blockquote>
<p>UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  …\torch\csrc\utils\tensor_numpy.cpp:180.)</p>
</blockquote>
<p>My specific use case it that I load a block of bytes from disk and want to get that into a pytorch tensor. Here is a simplified example:</p>
<pre data-code-wrap="python"><code class="lang-python"># just write some dummy data
with open("data.txt", "wb") as f:
    f.write(b"test")

# I want to get the file data into this tensor in only a single copy
result = torch.empty(1024)

with open("data.txt", "rb") as f:
    buffer: bytes = f.read()
    array: np.array = np.frombuffer(buffer, dtype=np.float32)
    result[0:4] = torch.from_numpy(array)
</code></pre>
<p>The warning triggers on the last line of my example code. Of course it is right, and in general this is a dangerous thing to do, but in this specific case the wrapped tensor is only used to get data into a real tensor and then immediately discarded.</p>
<p>Just removing <code>torch.from_numpy</code> results in the following error:</p>
<blockquote>
<p>TypeError: can’t assign a numpy.ndarray to a torch.FloatTensor</p>
</blockquote>
<p>Is there a way to directly copy data from numpy into PyTorch without triggering the original warning? I know I can disable the warning but that’s unsafe since I might make this mistake for real somewhere else in the codebase.</p>
<p>Thanks!</p>