akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Is there any work around to make a differentiable torch.max?

Alex_Luya: <p>(I have read this:<a href="https://discuss.pytorch.org/t/got-no-graph-nodes-that-require-computing-gradients-when-use-torch-max/7952,but">https://discuss.pytorch.org/t/got-no-graph-nodes-that-require-computing-gradients-when-use-torch-max/7952,but</a> it isn’t what I want to find)</p>
<p>Hello,in a segmentation work,I have<br>
<code>logits  BxCxHxW    and   targets  BxHxW</code><br>
and want to calculate<br>
<code>TP,FP,TN,FN</code><br>
so I did something like this:</p>
<pre><code class="lang-auto">        normalizeds = F.softmax(logits) 
        _, preds = torch.max(normalizeds, 1)   #&lt;---torch.max NOT differentiable
</code></pre>
<p>then using code like:</p>
<pre><code class="lang-auto">    for clazz in range(logits.shape[1]):        #loop by channels(class actually)
            predict_true = preds == clazz       #get predicted truths of specific class
            real_true = targets == clazz          #get real truths of specific class
            TP=real_true.squeeze()[preict_true.squeeze()].int().sum()   #calc true positive of specific class
</code></pre>
<p>The problem is that the ‘torch.max’ is NOT differentiable,any work around to make a differentiable torch.max,ultimately,calculate the TP,TN,FP,TN without losing differentiablity?</p>

Sayed_Nadim: <p>Please see this <a href="https://discuss.pytorch.org/t/confused-about-torch-max-and-gradient/14283">issue</a></p>
Speed is different with same functionality

yoko: <p>I tested to create cost volume(stereo depth estimation)<br>
then I tried to evaluate 2 method below<br>
what makes time that differ?</p>
<pre><code class="lang-auto">        start_full_time = time.time()
	B, C, H, W = refimg_fea.shape
	cost = refimg_fea.new_zeros([B, 2*C, self.maxdisp//4, H, W], requires_grad=False)
        print('time = %.4f [s]' %((time.time() - start_full_time)))
	print(cost.device)
</code></pre>
<p>time = 0.0000 [s]<br>
cuda:0</p>
<pre><code class="lang-auto">        start_full_time = time.time()
	B, C, H, W = refimg_fea.shape
	cost = torch.FloatTensor(B, C*2, self.maxdisp//4,  H,  W).cuda()
        print('time = %.4f [s]' %((time.time() - start_full_time)))
	print(cost.device)
</code></pre>
<p>time = 0.1875 [s]<br>
cuda:0</p>
<p>my guess is first code is createing tensor in cuda<br>
secound code is creating tensor in cpu and move to cuda<br>
so moving to cuda consuming operation time</p>
<p>if yes, is there any way to directly create tensor in cuda?</p>

ptrblck: <p>The reasons could be different code paths taken by the tensor constructors.<br>
However, since CUDA operations are asynchronous, you would need to synchronize the code before starting and stopping the timer.<br>
The <code>torch.utils.benchmark</code> utilities can be helpful, as they are synchronizing and add warmup iterations.</p>

yoko: <p>thank you for your reply</p>
<p>Let me confirm<br>
what you are saying is timer is CPU and tensor process is GPU, so it doesn’t synchronize. that’s why I need to synchronize?</p>
<p>thank you in advance</p>

ptrblck: <p>Yes, you would be profiling the CPU calls and kernel launch, not the actual data transfer or any workload on the GPU as it’s executed asynchronously and the CPU can run ahead and stop the timer, while the GPU is still busy.</p>
RuntimeError: "lu_cuda" not implemented for 'Half'

AlphaBetaGamma96: <p>Hi All,</p>
<p>I’ve been starting to run my code on a GPU and started to change the default dtype via <code>torch.set_default_type(torch.half)</code>. However, because my model uses a <code>torch.slogdet</code> function within it. It doesn’t seem to work, I assume because <code>torch.half</code> isn’t supported by <code>torch.slogdet</code>? (Or is to do with LU decomposition that might be happening under the hood within <code>torch.slogdet</code>?)</p>
<pre><code class="lang-auto">RuntimeError: "lu_cuda" not implemented for 'Half'
</code></pre>
<p>Is there a potential work around for this? Or is this something that’ll be added within the future?</p>
<p>Thank you in advance!</p>

ptrblck: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="1" data-topic="110389">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>Is there a potential work around for this? Or is this something that’ll be added within the future?</p>
</blockquote>
</aside>
<p>Linear algebra methods are usually unstable in reduced precision, so you should use <code>float32</code> for these kind of operations.<br>
I don’t think <code>float16</code> support will be added to <code>lu_cuda</code> for the aforementioned stability reasons.</p>

AlphaBetaGamma96: <p>Thanks for the response and explanation!</p>
<p>Do you think it might be possible to use the automatic mixed-precision module to get any speed benefits with the linear layers I used but keep <code>torch.slogdet</code> as float32?</p>
<p>Thanks!</p>

ptrblck: <p>Yes, that should be possible. The automatic-mixed precision <code>autocast</code> should use <code>float32</code> for <code>slogdet</code>, if I’m not mistaken, but you would most likely need to remove the usage of <code>set_default_type</code>.</p>
Using Transformer Module for time series?

isaacmg: <p>Hi I’m using the PyTorch transformer module for time series forecasting and I have a couple questions related to the <code>tgt</code> sequence as well as few more general questions. (i.e the module from <code>from torch.nn.modules import Transformer</code>). For the transformer I’m aware that we generally feed in the actual target sequence (as opposed to generating the target sequence step by step like other encoder-decoder methods). So my first question is prior to the transformer I have a standard linear layer to transform my time series sequence along with positional encodings:</p>
<pre><code class="lang-auto">class TransformerTimeSeries(torch.nn.Module)
  def __init__(self, n_time_series, d_model=128):
        super().__init__()
        self.dense_shape = torch.nn.Linear(n_time_series, d_model)
        self.pe = SimplePositionalEncoding(d_model)
        self.transformer = Transformer(d_model, nhead=8)
</code></pre>
<p>As per the transformer module code the src and trg sequence need to be the same dimension. So I was wondering can I simply do something like this or will this somehow leak information about the target?</p>
<pre><code class="lang-auto">def forward(self, x, t):
        x = self.dense_shape(x)
        x = self.pe(x)
        t = self.dense_shape(t)
        t = self.pe(t)
        x = self.transformer(x, t)
</code></pre>
<p>Second question: Does the target sequence need an offset? So for instance if I have the time series [0,1,2,3,4,5,6,7] and I want to feed in [0,1,2,3] to predict [4,5,6,7] (tgt)? Would I simply feed it in like that or is it more complicated? Final question is will I need a mask for the encoder as well? My inclination is yes as unlike with sentence I would want the current timestep to be formed solely by the previous ones.</p>
<p>Thanks for the help.</p>

isaacmg: <p>Anyone have links to an example of one that uses the full decoder? I’m currently having a problem of getting a lot of NaN values in the output using this approach.</p>

shamoons: <p>I’m wondering if you’ve had any luck with this yet?</p>

isaacmg: <p>Yeah I’ve it up and running for a while. You can see the implementation at <a href="https://github.com/AIStream-Peelout/flow-forecast" class="inline-onebox" rel="noopener nofollow ugc">GitHub - AIStream-Peelout/flow-forecast: Deep learning PyTorch library for time series forecasting, classification, and anomaly detection (originally for flood forecasting).</a></p>
How to convert torch::conv2d return value(tensor) to cv::mat

yzqxmu: <p>I run the following program:</p>
<p>read a picture of 3 channel input torch::nn::conv2d(3,3,3).pad(1).stride(1),then I got the results:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48db443ae3aaefacf943ac2cef9189c7ff2b40a9.jpeg" alt="results" data-base62-sha1="aow9yRVtTwrvS8uceiV19N1iFZv" width="256" height="256"><br>
code:</p>
<pre><code class="lang-auto">cv::Mat img = cv::imread("babyx2.png", 1);
torch::Tensor img_tensor = torch::from_blob(img.data, { img.rows, img.cols, 3 }, torch::kByte);
img_tensor = img_tensor.permute({ 2, 0, 1 }); 
img_tensor = img_tensor.unsqueeze(0);
img_tensor = img_tensor.to(kFloat32);
torch::Tensor result = C1(img_tensor);       //C1: torch::nn::Conv2d(torch::nn::Conv2dOptions(3, 3, 5).padding(1))
.....then get the result use following method
auto ToCvImage(at::Tensor tensor)
{
	int width = tensor.sizes()[0];
	int height = tensor.sizes()[1];
	//auto sizes = tensor.sizes();
	try
	{
		cv::Mat output_mat(cv::Size{ height, width }, CV_8UC3, tensor.data_ptr&lt;uchar&gt;());

		return output_mat.clone();
	}
	catch (const c10::Error&amp; e)
	{
		std::cout &lt;&lt; "an error has occured : " &lt;&lt; e.msg() &lt;&lt; std::endl;
	}
	return cv::Mat(height, width, CV_8UC3);
}
</code></pre>
<p>what happen???I don’t think the result picture is right,how to do…help</p>

yzqxmu: <p>I input picture following:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cbde6d1c2c2ad6e7a06bdf337ebb3f8256ee5c8.jpeg" alt="babyx2" data-base62-sha1="aWT9j7YD6xVs2kPU3UVEWf2toU0" width="256" height="256"><br>
then torch::nn::Conv2D(3,3,5).padding.stride<br>
why the return value of torch::nn::Conv2D like that??? How to show it in the right way？？</p>

ptrblck: <p>You are permuting the tensor from channels-last to channels-first format here:</p>
<pre><code class="lang-python">img_tensor = img_tensor.permute({ 2, 0, 1 }); 
</code></pre>
<p>and I don’t see the permutation back to channels-last, which is the expected memory layout in OpenCV.<br>
If you pass a channels-first image to OpenCV, the result would be interleaved, which fits your first posted image.</p>

yzqxmu: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="110367">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You are permuting the tensor from channels-last to channels-first format here:</p>
</blockquote>
</aside>
<p>Thank you. It worked well</p>
GRU batch_first changing between layers

Backup_Backup: <p>Hi,</p>
<p>while reading about the ASR project implementation here <a href="https://www.assemblyai.com/blog/end-to-end-speech-recognition-pytorch" class="inline-onebox" rel="noopener nofollow ugc">Building an end-to-end Speech Recognition model in PyTorch</a> I came across a GRU implementation that is unlike any other RNN/GRU/LSTM I have come across.</p>
<p>The reason why I am curious is that this implementation has outperformed every other network  I have tried in my experiments.</p>
<p>The implementation is as follows:</p>
<p>This is the GRU:</p>
<pre><code>class BidirectionalGRU(nn.Module):
    def __init__(self, rnn_dim, hidden_size, dropout, batch_first):
         super(BidirectionalGRU, self).__init__()

         self.BiGRU = nn.GRU(
             input_size=rnn_dim, hidden_size=hidden_size,
             num_layers=1, batch_first=batch_first, bidirectional=True)
        self.layer_norm = nn.LayerNorm(rnn_dim)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
       x = self.layer_norm(x)
       x = F.gelu(x)
       x, _ = self.BiGRU(x)
       x = self.dropout(x)
       return x
</code></pre>
<p>Then in the main network class the multilayer GRU is created as follows:</p>
<pre><code>self.birnn_layers = nn.Sequential(*[
                 BidirectionalGRU(rnn_dim=rnn_dim if i==0 else rnn_dim*2,
                                              hidden_size=rnn_dim, 
                                              dropout=dropout, 
                                              batch_first=i==0)
                                              for i in range(n_rnn_layers)
    ])
</code></pre>
<p>What I don’t understand is why the first layer has batch_first=True and then all subsequent layers make use of batch_first=False.</p>
<p>If anyone is familiar with why this is being done I would really appreciate any help.</p>

googlebot: <p>that’s done because output of the first GRU is time-major, despite batch first input</p>

Backup_Backup: <p>After looking at the docs I saw that the output is always time-major as you said. I wrote this test to confirm that this was what was going on</p>
<pre><code>gru_bf = nn.GRU(input_size=512,
             hidden_size=1024,
             num_layers=1,
             batch_first=True,
             bidirectional=False)

gru = nn.GRU(input_size=512,
             hidden_size=1024,
             num_layers=1,
             batch_first=False,
             bidirectional=False)

batch = 1
seq = 100
input_size = 512

in_bf = torch.randn(batch, seq, input_size)
input = torch.randn(seq, batch, input_size)

out_bf,_ = gru_bf(in_bf)
out, _ = gru(input)

print(out_bf.shape)
print(out.shape)
</code></pre>
<p>The output is:</p>
<pre><code>torch.Size([1, 100, 1024])
torch.Size([100, 1, 1024])
</code></pre>
<p>It seems that if batch_first is true the output will also have batch first. Is this indeed the case or am I misinterpreting the output.</p>
<p>Thank you for your help I really appreciate it.</p>