akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchscript states device mismatch

GeorgePearse: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c6f4a12d669ae9a251ecedc47a13ce02606efbd.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1c6f4a12d669ae9a251ecedc47a13ce02606efbd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c6f4a12d669ae9a251ecedc47a13ce02606efbd_2_690x468.jpeg" alt="image" data-base62-sha1="43xNWmqSYCeHUiEdlQlLQmG7OnX" width="690" height="468" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c6f4a12d669ae9a251ecedc47a13ce02606efbd_2_690x468.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c6f4a12d669ae9a251ecedc47a13ce02606efbd_2_1035x702.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c6f4a12d669ae9a251ecedc47a13ce02606efbd_2_1380x936.jpeg 2x" data-dominant-color="F2D9D8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1305 243 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m using the export_model.py script in Detectron2 and receive this error. I’ve definitely successfully exported models with this script before, and ran them on cuda, no idea what’s changed.</p>
<p>Wonder if the error message might be misleading?</p>

ptrblck: <p>I don’t know which model you are using, but maybe this error is related to <a href="https://discuss.pytorch.org/t/unable-to-trace-maskrcnn-model-to-torchscript/147415">this one</a> which also failed if the model was traced but I believe scripting worked.</p>
Processing embeddings and similarity scores - a better way to handle using tensors

alywonder: <p>I have a 100x100 tensor of similarity scores of 100 sentences I calculated. I also have a dictionary of 5 categories with a list of indexes showing which rows and columns in the similarity score tensor belong to each category.</p>
<pre><code class="lang-auto">scores.shape

&gt;&gt; torch.Size([100, 100])

cat_index

{ 'Album': [14, 31, 34, 87],
  'Animal': [21, 85, 99, 10],
  ...
  'Artist': [12, 15, 32, 45, 46, 48, ...]
}
</code></pre>
<p>I want to find the average score by category. That means I want to find the average by rows and columns based on the indexes for each category. The 100x100 tensor will be converted to 5x5.</p>
<p>I have done this by individually indexing and applying mean. But is there a better and more idiomatic way to do this with tensors? Will torch.nn.functional.embedding_bag() help?</p>
<p>Any help is much appreciated and thanks in advance.</p>
Lead Engineer ML & Data Science at JustWatch!

JustTJ: <h1>
<a name="lead-engineer-ml-data-science-for-justwatch-1" class="anchor" href="#lead-engineer-ml-data-science-for-justwatch-1"></a>Lead Engineer ML &amp; Data Science for JustWatch</h1>
<p>Location: Berlin, Germany, Full remote is possible +/-2 hours of CET) as is relocation</p>
<p>We own the world’s largest streaming guide, which gives us unparalleled data and insights on audience behaviors and content tastes. We understand entertainment audiences &amp; content better.</p>
<p>We deliver business outcomes, so if you’re a builder unafraid to challenge conventions and bring new ideas to the table, we want to hear from you! Bring our tailor-made tools for the needs of the movie industry to the very next level!</p>
<p>If you want to know much more about us: <a href="https://media.justwatch.com" rel="noopener nofollow ugc">media.justwatch.com</a></p>
<p>The complete job posting and application form is available here:<br>
<a href="https://jobs.lever.co/justwatch/a18c2c19-ab3e-40ba-b8a7-6a9c141412a5" rel="noopener nofollow ugc">https://jobs.lever.co/justwatch/a18c2c19-ab3e-40ba-b8a7-6a9c141412a5</a></p>
Torchserve - Custom handler for time-series forecasting - issue with response mismatched

DominiK_DK: <p>Hello</p>
<p>I would like to ask for help, advice and hints about issue I am facing when deploying NN model for time-series predictions with custom handler.<br>
I finally get working the model and inference for time-series prediction. Everything is working fine when i set prediction/output to 1.<br>
But what i need is to get it working for predicting more then 1 values. My model and handler provide these output but I’m getting some errors from Torchserve.</p>
<p>Response i get from curl is<br>
{<br>
“code”: 503,<br>
“type”: “InternalServerException”,<br>
“message”: “number of batch response mismatched”<br>
}<br>
In LOG from Trochserve i see following ( i put there some print for debug):<br>
Postprocess out:  [0.6994340419769287, 0.9235875606536865, 0.7007522583007812, 0.8973164558410645, 0.7195687294006348]<br>
[INFO ] W-9003-RNN_HR_model_1.0-stdout MODEL_LOG - model: RNN_HR_model, number of batch response mismatched, expect: 1, got: 5.</p>
<p>So i get 5 values in Inference.<br>
According to documentation for handlers i see there:</p>
<p><em>def inference(self, model_input):</em><br>
<em>:return: list of inference output in NDArray</em></p>
<p><em>def postprocess(self, inference_output):</em><br>
<em>:return: list of predict results</em></p>
<p>When i check my output i see that<br>
MODEL_LOG - The type of model output is  &lt;class ‘list’&gt;<br>
But even when i transform the inference output into NDArray and output for postprocess to list as provided in documentation I am still getting the same responses/errors</p>
<p>Any Idea what am i doing wrong? I can’t and dont want to use it in batch mode, since i have only 1 input and output should be 5 forecasted values.</p>
<p>Thank you</p>

marksaroufim: <p>You can wrap your output as a List of Lists so you’ll only return a List with a single element so just add an extra <code>[]</code></p>
<p>Another user faced a similar issue here <a href="https://github.com/pytorch/serve/issues/2169" class="inline-onebox" rel="noopener nofollow ugc">Return a list of dicts from prediction · Issue #2169 · pytorch/serve · GitHub</a></p>

DominiK_DK: <p>Great, it works!!!</p>
<p>Thank you very much for your help.<br>
Thank’s a lot.<br>
You really helped me a lot.</p>
Variable input length in each batch for custom word embedding

s_karki: <p>With the help of collate_fn i’ve created variable input length for each batch.(same input length in single batch but different in another)<br>
I want to create the sentiment classifier with the custom embedding.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class WordEmbedding(nn.Module):

    def __init__(self, vocab_size, embed_size):
        super(WordEmbedding, self).__init__()

        self.vocab_size = vocab_size
        self.embed_size = embed_size
        self.embedding = nn.Embedding(self.vocab_size, self.embed_size, sparse=True)
        self.fc1 = nn.Linear(self.embed_size*4767, 256)
        self.dropout = nn.Dropout(p=0.3)
        self.fc2 = nn.Linear(256, 1)
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        x = self.embedding(x).view((x.size(0), -1))     
        out = self.fc1(x)
        out = self.fc2(self.dropout(out))
        out = self.sigmoid(out)
        return out
</code></pre>
<p>Since, for each batch, there is different length(not equal to above vocab_size), there is matrix mis match error.<br>
I was wondering if there is any method i can train my model for different batch input length considering above model architecture.?</p>

Theo: <p>Did you use padding when tokenizing your dataset?</p>
Error in total_variation loss -- TypeError: list indices must be integers or slices, not tuple

shrbrh: <p>I have the following function for calculating total variation loss:</p>
<pre><code class="lang-auto">def tvLoss(self, img):
		vertical   = torch.abs(img[:, :, 1:, :] - img[:, :, :-1, :])
		horizontal = torch.abs(img[:, :, :, 1:] - img[:, :, :, :-1])
		loss       = torch.sum(horizontal) + torch.sum(vertical)
		return loss
</code></pre>
<p>But I get the following error:</p>
<pre><code class="lang-auto"> tv_loss  += loss.tvLoss(img)
  File "/home/project/network.py", line 169, in tvLoss
    vertical   = torch.abs(img[:, :, 1:, :] - img[:, :, :-1, :])
TypeError: list indices must be integers or slices, not tuple
</code></pre>
<p>How do I resolve this error?</p>

ptrblck: <p>I don’t know what your input is but would assume it’s a tensor.<br>
If so, then I cannot reproduce the error using:</p>
<pre><code class="lang-python">def tvLoss(img):
	vertical   = torch.abs(img[:, :, 1:, :] - img[:, :, :-1, :])
	horizontal = torch.abs(img[:, :, :, 1:] - img[:, :, :, :-1])
	loss       = torch.sum(horizontal) + torch.sum(vertical)
	return loss

img = torch.randn(2, 3, 24, 24)
out = tvLoss(img)
</code></pre>

shrbrh: <p>My input is of type  <code>&lt;class 'list'&gt;</code>.</p>

ptrblck: <p>Could you describe how this <code>list</code> should be indexed as it seems you expect it to have 4 dimensions?</p>

shrbrh: <p>It is actually the extracted features of an image. It is a list of 4 tensors extracted from 4 different layers of the VGG-19 network.</p>

ptrblck: <p>You would still need to index the list in a single dimension as seen here:</p>
<pre><code class="lang-python">l = [torch.randn(2) for _ in range(4)]
print(l)
# [tensor([ 0.3270, -2.2517]), tensor([-0.2995, -0.1264]), tensor([ 0.1325, -0.0489]), tensor([-1.1356, -0.3686])]

print(l[0])
# tensor([ 0.3270, -2.2517])

print(l[-1])
# tensor([-1.1356, -0.3686])

print(l[:, 0])
# TypeError: list indices must be integers or slices, not tuple
</code></pre>
Training time does not decrease by half even when doubling the number of GPUs

chopby: <p>Hello PyTorch Forum,</p>
<p>I am training a Transformer-based action recognition model and using Slurm for distributed processing. When I trained the model with 4 GPUs, it took 4 hours and 40 minutes per epoch. However, I expected the training time to be reduced by half, to 2 hours and 20 minutes, when I trained the model with 8 GPUs. However, the training time for each epoch is 3 hours and 30 minutes, which is even longer than half of the original time. I also doubled the CPU and memory along with the GPUs.</p>
<p>The issue I anticipate is the bottleneck in the video loader. However, I don’t understand why doubling both the CPU cores and GPU doesn’t cut the time in half.</p>
<p>Why is this happening? What should I consider to reduce the training time?</p>

ptrblck: <p>You could profile your code using the native profiler or e.g. Nsight Systems and compare both systems to check where the bottleneck in your training is and e.g. if the GPUs are starving due to a slow data loading or processing.</p>
Mobilenetv3 decoder(list of tensors input) export issue

WEIIEW97: <p>Hello,</p>
<p>There was an error: <code>"RuntimeError: output 1 (3[ CPULongType{} ]) of traced region did not have observable data dependence with trace inputs; this probably indicates your program cannot be understood by the tracer."</code> encountered when I was exporting my mobilenetv3 decoder to onnx.</p>
<p>I suspect it is due to my input type since it contains multiple scales of image(list of tensors).</p>
<p>Here is my <code>decoder</code> code snippet:</p>
<pre><code class="lang-python">class MobilenetV3DecoderV1(nn.Module):
    def __init__(
        self,
        num_ch_enc,
        scales=range(4),
        num_output_channels=1,
        use_skips=True,
        mode="mobilenet_v3_small",
    ) -&gt; None:
        super().__init__()

        self.num_output_channels = num_output_channels
        self.use_skips = use_skips
        self.upsample_mode = "nearest"
        self.scales = scales

        self.num_ch_enc = num_ch_enc
        if mode == "mobilenet_v3_large" or mode == "mobilenet_v3_large_quantize":
            self.num_ch_dec = np.array([16, 32, 64, 128, 256])
        elif mode == "mobilenet_v3_small":
            self.num_ch_dec = np.array([4, 12, 20, 40, 80])

        # decoder
        self.convs = OrderedDict()
        self.num_of_scales = len(scales)
        for i in range(self.num_of_scales, -1, -1):
            # upconv_0
            num_ch_in = self.num_ch_enc[-1] if i == self.num_of_scales else self.num_ch_dec[i + 1]
            num_ch_out = self.num_ch_dec[i]
            self.convs[("upconv", i, 0)] = ConvBlock(num_ch_in, num_ch_out)

            # upconv_1
            num_ch_in = self.num_ch_dec[i]
            if self.use_skips and i &gt; 0:
                num_ch_in += self.num_ch_enc[i - 1]
            num_ch_out = self.num_ch_dec[i]
            self.convs[("upconv", i, 1)] = ConvBlock(num_ch_in, num_ch_out)

        for s in self.scales:
            self.convs[("dispconv", s)] = Conv3x3(
                self.num_ch_dec[s], self.num_output_channels
            )

        self.decoder = nn.ModuleList(list(self.convs.values()))
        self.sigmoid = nn.Sigmoid()

    def forward(self, input_features):
        self.outputs = {}
        # decoder
        x = input_features[-1]
        for i in range(self.num_of_scales, -1, -1):
            x = self.convs[("upconv", i, 0)](x)
            # if i != 0:
            #     x = self.convs[("attentionconv", i)](x, [input_features[i-1]])
            # else:
            x = [upsample(x)]
            if self.use_skips and i &gt; 0:
                x += [input_features[i - 1]]
            x = torch.cat(x, 1)
            x = self.convs[("upconv", i, 1)](x)
            if i in self.scales:
                self.outputs[("disp", i)] = self.sigmoid(self.convs[("dispconv", i)](x))
        return self.outputs
</code></pre>
<p>Is there a way can export it to onnx correctly?<br>
Thanks in advance!</p>
What is pytorch recommended approach to ensure sequential reads while dataloading read

Gavrish_Prabhu: <p>I’m running my training on imagenet data and using the generic data loader <code>torchvision.datasets.ImageFolder</code> for my use case. For higher batch sizes the data loader with transformation is taking substantial time to fetch every batch compared to the overall compute time. I have tried to improve performance by scaling the <code>num_of_workers</code> to max possible for my machine.</p>
<p>Definitely need a better way of preprocessing the data and storing it in a fashion to make more sequential reads. I have found many examples of creating the custom Dataset implementation to tackle this approach. I’m looking for what would be the standard recommended aproach to tackle this case?</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This post</a> and generally the thread could be helpful as also e.g. <code>DALI</code> is discussed there which might speed up your data loading and processing in case the CPU is “overworked”.</p>
Error in dice loss from torchgeometry

alienshallowalchemy: <p>Hi there,</p>
<p>I am trying to implement a dice loss for multi-class segmentation using torchgeometry.losses.DiceLoss(), but I keep getting an error</p>
<pre><code class="lang-auto">
"return one_hot.scatter_(1, labels.unsqueeze(1), 1.0) + eps RuntimeError: index -1 is out of bounds for dimension 1 with size 3"
</code></pre>
<p>I am passing to the dice loss object the network output (torch tensor shape [1,3,388,388] dtype float32) and label (torch tensor [1,388,388] dtype int64) as in documentation</p>
<p>I even did a test:</p>
<pre><code class="lang-auto">pred_mask = torch.rand(1, 3, 256, 256)
gt_mask = torch.rand(1, 256, 256).to(torch.int64)

print(pred_mask.shape, pred_mask.dtype)print(gt_mask.shape, gt_mask.dtype)

dice_loss = tgl.dice_loss(pred_mask, gt_mask)

print("Dice Loss: ", dice_loss.item())
</code></pre>
<p>and it works, it prints the dice loss for the similar shapes and datatypes</p>
<p>What am I doing wrong in the above case with that error ?</p>
<p>Thanks in advance</p>

ptrblck: <p>Your code works fine but based on the error message it seems your target contains a negative index:</p>
<pre><code class="lang-python">gt_mask[0, 0, 0] = -1
dice_loss = tgl.dice_loss(pred_mask, gt_mask)
# RuntimeError: index -1 is out of bounds for dimension 1 with size 3
</code></pre>
<p>and thus fails.</p>

alienshallowalchemy: <p>Yes it has negative values. I need to check how label is read, must have some error when reading or converting to int64</p>
<p>Thank you very much for your help.</p>
How to make heat equation dimensionless

Ali_geo: <p>Dear community,</p>
<p>I am trying to make a Physics Informed Neural Network for the heat equation in 1D:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bec47698ed29c6d960b426f0c5d38d265d506998.png" alt="image" data-base62-sha1="rdBDGaqfK8yvWeNTCWaVCdwwPmo" width="151" height="83"></p>
<p>I tried the following code to make a loss function for PDE residual:</p>
<blockquote>
<p>def lossPDE(self,x_PDE):<br>
g = x_PDE.clone()<br>
g.requires_grad = True # Enable differentiation<br>
f = self.forward(g)<br>
f_x_t = torch.autograd.grad(f,g,torch.ones([g.shape[0],1]).to(device),retain_graph=True, create_graph=True)[0] # first derivative of time<br>
f_xx_tt = torch.autograd.grad(f_x_t,g,torch.ones(g.shape).to(device), create_graph=True)[0]<span class="hashtag-raw">#second</span> derivative of x<br>
f_t = f_x_t[:,[1]]<br>
f_xx = f_xx_tt[:,[0]]<br>
f = f_t - alpha * f_xx<br>
return self.loss_function(f,f_hat) # f_hat is a tensor of zeros and it minimizes the f value</p>
</blockquote>
<p>In the simulation I am making a one day simulation (86400 seconds) for a bar with the length of 1 meters. Now, I want to make my PDE dimensionless. How can I do it in pytorch? Which part of my code should be changed? The unit of alpha is already in <em>m2/s</em>. I can share my whole code but it would be hundreds lines of code.<br>
I very much appreciate any help in advance.</p>
The results of training models using the PyTorch C++ API and PyTorch (Python) are significantly different

ZhiweiLuo: <p>Hi, I traced a custom model using the method torch.jit.script. However, when fine-tuning the torchscript on a dataset, I obtained different accuracy results (85% using PyTorch Python and 75% using PyTorch C++ API).</p>
<pre data-code-wrap="C++"><code class="lang-plaintext"># c++ 
torch::jit::script::Module model = torch::jit::load(options.model_path, torch::kCUDA); 
</code></pre>
<pre data-code-wrap="Python"><code class="lang-plaintext"># Python
model = torch.jit.load(model_path)
</code></pre>
<p>Also, I think that the data processing and optimization settings in PyTorch C++ API and PyTorch Python are the same. Because I was able to achieve the same results (acc 82%) using the method torch.jit.trace.Can anyone help?</p>

ZhiweiLuo: <p>Sorry, it was my mistake. The reason for the difference in accuracy between the PyTorch C++ API and Python is because I made an error in the data preprocessing step of the C++ version.</p>
ImportError: cannot import name 'efficientnet_v2_s' from 'torchvision.models'

ANS_CENTER: <p>Error ImportError: cannot import name ‘efficientnet_v2_s’ from ‘torchvision.models’ (/opt/conda/lib/python3.8/site-packages/torchvision/models/<strong>init</strong>.py)</p>

ptrblck: <p><code>efficientnet_v2_s</code> was added ~1 year ago in <a href="https://github.com/pytorch/vision/pull/5450">this PR</a> so you might need to update <code>torchvision</code> as I can properly import this model.</p>
Apply tensor-valued function rowwise to get another tensor

KiwiPower: <p>I have got a tensor of size <code>(N,2)</code></p>
<pre><code class="lang-auto">N = 10
t = torch.random(N,2)
</code></pre>
<p>and a function <code>f</code> that takes two values <code>x</code> and <code>y</code> and returns a (2,2) tensor, say</p>
<pre><code class="lang-auto">def f(x,y):
    return torch.tensor([[x,x+y],[x**2,y**3]])
</code></pre>
<p>How can I apply the function to each of the <code>N</code> rows of the first tensor and store the result as a <code>(N,2,2)</code> tensor? Thank you.</p>

ptrblck: <p>I’m unsure if I understand your use case correctly, but I assume you want to get the results of <code>x=t[0, 0]</code> and <code>y=t[0, 1]</code> for all “rows” of <code>t</code>.<br>
If so, you could use <code>torch.vmap</code> and rewrite the function a bit, as it will raise an error if a new tensor is created.<br>
This might work:</p>
<pre><code class="lang-python">def f(x):
    return torch.stack((
        torch.stack((x[0], x[0] + x[1])),
        torch.stack((x[0]**2, x[1]**2))),
        dim=0)


batched_f = torch.vmap(f)
batched_f(t)
# tensor([[[-1.8799e+00, -2.0183e+00],
#          [ 3.5340e+00,  1.9150e-02]],

#         [[-3.4007e-01, -2.0362e-01],
#          [ 1.1565e-01,  1.8621e-02]],

#         [[ 1.0567e-01,  1.6185e-01],
#          [ 1.1167e-02,  3.1562e-03]],

#         [[ 4.5812e-01,  4.7626e-01],
#          [ 2.0988e-01,  3.2880e-04]],

#         [[ 7.6334e-01,  7.1346e-01],
#          [ 5.8269e-01,  2.4886e-03]],

#         [[-1.9683e-01,  1.4371e+00],
#          [ 3.8741e-02,  2.6697e+00]],

#         [[ 1.1249e+00, -5.1169e-01],
#          [ 1.2654e+00,  2.6784e+00]],

#         [[-3.8157e-02, -1.0671e-01],
#          [ 1.4559e-03,  4.6996e-03]],

#         [[ 1.2200e+00,  2.6654e+00],
#          [ 1.4884e+00,  2.0891e+00]],

#         [[ 7.1752e-01, -1.5422e-01],
#          [ 5.1483e-01,  7.5992e-01]]])

for t_ in t:
    print(f(t_))
# tensor([[-1.8799, -2.0183],
#         [ 3.5340,  0.0192]])
# tensor([[-0.3401, -0.2036],
#         [ 0.1157,  0.0186]])
# tensor([[0.1057, 0.1619],
#         [0.0112, 0.0032]])
# tensor([[4.5812e-01, 4.7626e-01],
#         [2.0988e-01, 3.2880e-04]])
# tensor([[0.7633, 0.7135],
#         [0.5827, 0.0025]])
# tensor([[-0.1968,  1.4371],
#         [ 0.0387,  2.6697]])
# tensor([[ 1.1249, -0.5117],
#         [ 1.2654,  2.6784]])
# tensor([[-0.0382, -0.1067],
#         [ 0.0015,  0.0047]])
# tensor([[1.2200, 2.6654],
#         [1.4884, 2.0891]])
# tensor([[ 0.7175, -0.1542],
#         [ 0.5148,  0.7599]])
</code></pre>
Error in installing Pytorch version 1.4.0 in Google Colab

dena_shamsollahi: <p>I need to downgrade the Pytorch, when I use these two commands in Google colab, I face the following error.<br>
!pip install torchvision==0.5.0<br>
!pip install torch==1.4.0</p>
<p>Looking in indexes: <a href="https://pypi.org/simple" class="inline-onebox" rel="noopener nofollow ugc">Simple index</a>, <a href="https://us-python.pkg.dev/colab-wheels/public/simple/" rel="noopener nofollow ugc">https://us-python.pkg.dev/colab-wheels/public/simple/</a><br>
ERROR: Could not find a version that satisfies the requirement torchvision==0.5.0 (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.8.2, 0.9.0, 0.9.1, 0.10.0, 0.10.1, 0.11.0, 0.11.1, 0.11.2, 0.11.3, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.14.1)<br>
ERROR: No matching distribution found for torchvision==0.5.0<br>
ERROR: Could not find a version that satisfies the requirement torch==1.4.0 (from versions: 1.7.1, 1.8.0, 1.8.1, 1.9.0, 1.9.1, 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.12.0, 1.12.1, 1.13.0, 1.13.1)<br>
ERROR: No matching distribution found for torch==1.4.0</p>

ptrblck: <p>Based on <code>sys.version</code> Colab is using <code>Python==3.9.16</code>, which wasn’t supported when PyTorch <code>1.4.0</code> was released.<br>
The supported Python versions are 2.7, 3.5, 3.6, 3.7, and 3.8 as seen <a href="https://pypi.org/project/torch/1.4.0/#files">here</a>.<br>
What’s the reason you want to download such as old PyTorch release?</p>
Backpropagate through part of input data

deponce: <p>Hi there,</p>
<p>Now I’m working on a project that needs to generate hundreds of adversarial copies for one batch on ImageNet. In terms of saving memory and running time, we want to only add perturbation(calculate gradient) on the images that are not correctly classified. Is it possible to implement it in torch?</p>
<p>Thanks in advance!</p>
Torch.load ordereddict

woon: <p>I want to my pretrained resnet32.pth to resnet32.wts for quantization</p>
<p>resnet50.pth provided by torchvision</p>
<p>net = torch.load(‘resnet50.pth’)<br>
net = net.to(‘cuda:0’)<br>
…</p>
<p>is working but resnet32.pth that i trained by me isn’t working</p>
<p>net = torch.load(‘resnet32.pth’)<br>
net = net.to(‘cuda:0’)<br>
…</p>
<p>AttributeError: ‘collections.OrderedDict’ object has no attribute ‘to’</p>
<p>resnet50.pth is resnet and resnet32.pth is ordereddict so i want to change resnet32.pth to resnet</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba82e7fc28abaf86d1dfff3c83516201393aabd8.png" data-download-href="https://discuss.pytorch.org/uploads/default/ba82e7fc28abaf86d1dfff3c83516201393aabd8" title="resnet50"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba82e7fc28abaf86d1dfff3c83516201393aabd8_2_690x101.png" alt="resnet50" data-base62-sha1="qBXh22G8m882jXptWwPIQkOVPG8" width="690" height="101" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba82e7fc28abaf86d1dfff3c83516201393aabd8_2_690x101.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/ba82e7fc28abaf86d1dfff3c83516201393aabd8_2_1035x151.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/a/ba82e7fc28abaf86d1dfff3c83516201393aabd8.png 2x" data-dominant-color="A4A5A9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resnet50</span><span class="informations">1333×197 23.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>resnet32.pth</code> seems to be a <code>state_dict</code> so you would need to either create the model object beforehand using its actual source code before loading this <code>state_dict</code> or you could try to save the entire model (as was apparently done for <code>resnet50</code>), which can fail in multiple ways if you don’t guarantee to restore the source tree on the machine trying to load the model.</p>
How to register new backend

ar51: <p>Hi All,<br>
I’m new to the pytorch2.0 flow. I want to register a new backend to the torch. Compile and get control to aten/prim IR, is there any documentation available for this flow or any sample toy backend kind?</p>

marksaroufim: <p>Looks like you got an answer here so posting for others <a href="https://dev-discuss.pytorch.org/t/registering-new-compiler-backend-in-pytorch2-0/1092/4" class="inline-onebox" rel="noopener nofollow ugc">Registering new compiler backend in Pytorch2.0 - #4 by jansel - compiler - PyTorch Dev Discussions</a></p>
<p>And this tutorial might help <a href="https://pytorch.org/docs/master/dynamo/custom-backends.html" class="inline-onebox" rel="noopener nofollow ugc">Custom Backends — PyTorch master documentation</a></p>
What is the recommend serialization format when considering the upcoming pt2?

yushu.gao: <p>There are two serialization format: torchscript and onnx.</p>
<p>torchscript seems will be deprecated in the long term: <a href="https://dev-discuss.pytorch.org/t/the-nuances-of-pytorch-graph-capture/501/9?u=shuokay" class="inline-onebox" rel="noopener nofollow ugc">The nuances of PyTorch Graph Capture - #9 by penguinwu - compiler - PyTorch Dev Discussions</a>, and after using torchscript for a period of time, we found that torchscript is not easy to use.</p>
<p>ONNX has a relatively larger number of users, but onnx can’t be loaded and run by torch directly, which lead to potential numeric issues.</p>
<p>My question is: is there a new serialization format in pt2? If not, which format you guys recommend for model serialization?</p>

marksaroufim: <p>We’re still working on this but I’ll reach out again in a few months to provide an update if possible</p>
Is torch.float16 different from IEEE754? torch.float16 shows smaller value than 2**(-15)

Marysan: <p>I noticed that torch.float16’ s smallest positive number 2**(-24). However, IEEE 754 set the smallest non-zero positive number as 2**(-14). (when not considering denormal representation)</p>
<p>How does torch.float16 work? Is it different from IEEE754 ? Do they apply different exponent bias (a.k.a. zero offset) according to the tensor’s distribution?</p>
<p>The below code shows what I have tried:</p>
<blockquote>
<blockquote>
<blockquote>
<p>x = torch.tensor((2**(-24)*1.))<br>
x<br>
tensor(5.9605e-08)<br>
x.half()<br>
tensor(5.9605e-08, dtype=torch.float16)</p>
</blockquote>
</blockquote>
</blockquote>

KFrank: <p>Hi Mary!</p>
<aside class="quote no-group" data-username="Marysan" data-post="1" data-topic="174423" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marysan/48/58144_2.png" class="avatar"> Marysan:</div>
<blockquote>
<p>How does torch.float16 work?</p>
</blockquote>
</aside>
<p>You have it right.</p>
<aside class="quote no-group">
<blockquote>
<p>Is it different from IEEE754 ?</p>
</blockquote>
</aside>
<p>No.  <code>torch.float16</code> is standard IEEE 754.  (It actually depends on your<br>
hardware, but for all practical purposes, all* hardware is IEEE 754.)</p>
<p>Your only problem is that the your example of <code>2**-24</code> gives your a<br>
subnormal number (as per IEEE 754).</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.1
&gt;&gt;&gt; import numpy as np
&gt;&gt;&gt;
&gt;&gt;&gt; torch.finfo (torch.half).smallest_normal
6.103515625e-05
&gt;&gt;&gt; 2**-14
6.103515625e-05
&gt;&gt;&gt;
&gt;&gt;&gt; torch.tensor (5.9605e-08).half()
tensor(5.9605e-08, dtype=torch.float16)
&gt;&gt;&gt; 2**-24
5.960464477539063e-08
&gt;&gt;&gt;
&gt;&gt;&gt; torch.tensor (torch.finfo (torch.half).smallest_normal, dtype = torch.half)
tensor(6.1035e-05, dtype=torch.float16)
&gt;&gt;&gt; torch.tensor (torch.finfo (torch.half).smallest_normal, dtype = torch.half) / 2
tensor(3.0518e-05, dtype=torch.float16)
&gt;&gt;&gt;
&gt;&gt;&gt; torch.tensor (5.9605e-08).half()
tensor(5.9605e-08, dtype=torch.float16)
&gt;&gt;&gt; torch.tensor (5.9605e-08).half() / 2
tensor(0., dtype=torch.float16)
&gt;&gt;&gt;
&gt;&gt;&gt; np.finfo (np.half).smallest_normal
6.104e-05
&gt;&gt;&gt; np.finfo (np.half).smallest_subnormal
6e-08
</code></pre>
<p>*)  Well, there is always NVIDIA’s dreaded TensorFloat-32 format …</p>
<p>Best.</p>
<p>K. Frank</p>
My PyTorch is not working

samuele: <p>Hi, I’m writing this message as I saw all the others but they didn’t help me, I hope there will be someone here who understands the problem. I tried to download PyTorch in all the different ways, by creating new environments, etc, but it still not working. If you need any photos just ask me. Thanks in advance<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/9/e9276485400de697198e4930a26acce226c7d572.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e9276485400de697198e4930a26acce226c7d572" title="Schermata 2023-03-09 alle 19.35.04"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9276485400de697198e4930a26acce226c7d572_2_690x431.jpeg" alt="Schermata 2023-03-09 alle 19.35.04" data-base62-sha1="xgzEeJPWzgruH5GjgL4oPTB0X5w" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9276485400de697198e4930a26acce226c7d572_2_690x431.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9276485400de697198e4930a26acce226c7d572_2_1035x646.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/9/e9276485400de697198e4930a26acce226c7d572_2_1380x862.jpeg 2x" data-dominant-color="272529"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Schermata 2023-03-09 alle 19.35.04</span><span class="informations">1920×1200 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
React Native 0.71.0 requires fbjni version 0.3.0

SomaKishimoto: <p>We’re having issues with PlayTorch, which uses PyTorch Android as an internal dependency.<br>
The problem is that <a href="https://github.com/facebookresearch/playtorch/issues/195#issuecomment-1426637620" rel="noopener nofollow ugc">PyTorch</a> is still using fbjni version 0.2.2, while React Native 0.71.0 requires version 0.3.0.<br>
We’ve provided more details in this issue:</p>
<ul>
<li><a href="https://github.com/facebookresearch/playtorch/issues/195#issuecomment-1426637620" class="inline-onebox" rel="noopener nofollow ugc">React Native 0.71 compatibility is required · Issue #195 · facebookresearch/playtorch · GitHub</a></li>
</ul>
<p>We’re wondering what possible solutions can be considered and would appreciate any advice.</p>
<p>Thank you.</p>
Machine translation in just transformer-decoder

Ollie_J: <p>I’m solving a rare machine translation problem, where I map a robotic piece of music (imagine each note played for the exact duration + volume) to an actual human performance of such piece. I was hoping to use a transformer encoder-decoder architecture, but tokenization gives sequences with 15,000 tokens on average, which is too computationally expensive to manipulate with my budget.</p>
<p>However, all of the tokenized examples have the input robotic piece almost the same length as the output performance. Could I just use the transformer decoder architecture to map between the two, and use special tokens to pad to ensure they’re the same length?</p>
<ul>
<li>e.g. if len(input) &lt; len(output), pad input to output length with  tokens</li>
<li>e.g. if len(input) &gt; len(output), pad output to input length with  tokens</li>
</ul>
How to make some slight changes to the convolutional layer in neural network?

yosiki.life: <p>I want to make some small changes in the convolutional layer of my neural network model, that is, to replace the input pixels at the center of the calculation and then perform convolution. At the same time, I also want to be able to continue to use the GPU for training. How should I do it? (sorry for my poor English)</p>

ptrblck: <p>I’m unsure if you want to manipulate the input activation or the actual conv filter.<br>
These approaches should work and you can pick which fits your use case:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 16, 3, 1, 1).cuda()
x = torch.randn(1, 1, 24, 24, requires_grad=True, device="cuda")

# calculate fake activation output 
out = x * 2
# manipulate any pixels
out[:, :, 12, 12] = 1.0

out = conv(out)


x = torch.randn(1, 1, 24, 24, requires_grad=True, device="cuda")
# manipulate first filter
with torch.no_grad():
    conv.weight[0, 0, 1, 1] = 1.
out = conv(x)
</code></pre>

yosiki.life: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="174407">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m unsure if you want to manipulate the input activation or the actual conv filter.<br>
These approaches should work and you can pick which fits your use case:</p>
</blockquote>
</aside>
<p>What I mean is to replace the central input pixel within the range of the convolution window, not the center of the entire input, such as replacing the average value of the surrounding pixels in the convolution window, because I want to do this for every pixel of the input , I don’t want to use the center input pixel within the convolution window range when convolving, but use the average value of the neighbors instead, it may not be clear, sorry</p>

ptrblck: <p>If you want to implement a mean filter, you could directly set the filter weights via:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 1, 3, bias=False)
with torch.no_grad():
    conv.weight.fill_(1./9)
    
x = torch.arange(7*7).view(1, 1, 7, 7).float()
print(x)
# tensor([[[[ 0.,  1.,  2.,  3.,  4.,  5.,  6.],
#           [ 7.,  8.,  9., 10., 11., 12., 13.],
#           [14., 15., 16., 17., 18., 19., 20.],
#           [21., 22., 23., 24., 25., 26., 27.],
#           [28., 29., 30., 31., 32., 33., 34.],
#           [35., 36., 37., 38., 39., 40., 41.],
#           [42., 43., 44., 45., 46., 47., 48.]]]])

out = conv(x)
print(out)
# tensor([[[[ 8.0000,  9.0000, 10.0000, 11.0000, 12.0000],
#           [15.0000, 16.0000, 17.0000, 18.0000, 19.0000],
#           [22.0000, 23.0000, 24.0000, 25.0000, 26.0000],
#           [29.0000, 30.0000, 31.0000, 32.0000, 33.0000],
#           [36.0000, 37.0000, 38.0000, 39.0000, 40.0000]]]],
#        grad_fn=&lt;ConvolutionBackward0&gt;)
</code></pre>
Models for 2d(N x M) matrix inputs and (N x 1) outputs

Neilddd: <p>I have many (N x M) matrix as features and (N x 1) matrix as the lables.<br>
I want to train a model, the input is (N x M) matrix, and the model can predict the (N x 1) matrix.<br>
Assume each row contains features of an item, so the matrix contains N items, the labels are the probabilities for each item.</p>
<p>Any advices? How to implement this model?</p>
If the model is trained by mixed precision and saved by .half(), are there any accuracy losses between such model and its TRT/ONNX conversion?

sakura: <p>Hi, we know that there are inevitable accuracy losses between the trained model (FP32 or TF32) and its FP16 deployment conversions, like TRT/ONNX.</p>
<p>When it comes to mixed precision training, I am wondering that could such trained FP16 model be able to perfectly align with its .half() saved model and moreover, its FP16 deployments?</p>
<p>Thanks : )</p>
Precision, Acuuracy not increasing on binary Timeseries classifer

granth_jain: <p>Hi,</p>
<p>I am unable to increase the precision and recall on my time-series classifier in below code.<br>
Can someone please help to let me know what can be done further to improve the precision and recall.</p>
<pre><code>class lstmfcn(nn.Module):
	def __init__(self, input_shape, size):
		super(lstmfcn, self).__init__()
		self.lstm = nn.LSTM(input_size=input_shape[1], hidden_size=size, batch_first=True)
		self.conv1d_1 = nn.Conv1d(input_shape[0], size, kernel_size=8, padding='same')
		self.batch_norm_1 = nn.BatchNorm1d(size)
		self.activation_1 = nn.ReLU()
		self.conv1d_2 = nn.Conv1d(size, size*2, kernel_size=5, padding='same')
		self.batch_norm_2 = nn.BatchNorm1d(size*2)
		self.activation_2 = nn.ReLU()
		self.conv1d_3 = nn.Conv1d(size*2, size, kernel_size=3, padding='same')
		self.batch_norm_3 = nn.BatchNorm1d(size)
		self.activation_3 = nn.ReLU()
		self.global_avg_pooling_1d = nn.AdaptiveAvgPool1d(1)
		self.fc = nn.Linear(size*2, 1)
		self.sigmoid = nn.Sigmoid()

	def forward(self, x):
		h_lstm, _ = self.lstm(x)
		y = self.conv1d_1(x)
		y = self.batch_norm_1(y)
		y = self.activation_1(y)
		y = self.conv1d_2(y)
		y = self.batch_norm_2(y)
		y = self.activation_2(y)
		y = self.conv1d_3(y)
		y = self.batch_norm_3(y)
		y = self.activation_3(y)
		y = self.global_avg_pooling_1d(y)
		x = torch.cat((h_lstm[:, -1], y.squeeze(dim=2)), dim=1)
		x = self.fc(x)
		x = self.sigmoid(x)
		return x
</code></pre>
<p>Has already done scaling and resampling.<br>
Increasing the size is not helping.</p>

ptrblck: <p>Based on your model definition it seems you are using <code>torch.sigmoid</code> + <code>nn.BCELoss</code>.<br>
Could you remove the <code>torch.sigmoid</code> usage and use <code>nn.BCEWithLogitsLoss</code> instead for a better numerical stability and check if this would help the model training?</p>

granth_jain: <p>Hi,<br>
I already tried that, the model loss decreases but the precision and recall remains the same.</p>

granth_jain: <p>After adding dropout, the minimum val_loss has decreased, but not by much.</p>
I having an error in my data vusalization code in pytorch?

Teddy_Salas: <p>(‘’‘’'I am having an error in data visualization code,this is the code:<br>
import torch<br>
from torch import nn<br>
import matplotlib.pyplot as plt</p>
<p>weight =0.7<br>
bias=0.3<br>
X = torch.arange(0,3,0.02)<br>
Y = weight * X * bias<br>
Training_Set = int(0.8 * len(X))<br>
Training_SetofX = len(X[:Training_Set])<br>
Testing_SetofX = len(X[Training_Set:])<br>
Training_SetofY = len(Y[:Training_Set])<br>
Testing_SetofY = len(Y[Training_Set:])</p>
<p>print(f"Training set of X {len(X[:Training_Set])}“)<br>
print(f"Testing set of X {len(X[Training_Set:])}”)<br>
print(f"Training set of Y {len(Y[:Training_Set])}“)<br>
print(f"Testing set of Y {len(Y[Training_Set:])}”)</p>
<p>def plot_predictions(train_data = Training_SetofX,<br>
train_label = Training_SetofY,<br>
test_data = Testing_SetofX,<br>
test_label = Testing_SetofY,<br>
predictions = None):<br>
plt.figure(figsize = (10,7))<br>
plt.scatter(train_data,train_label,c=“b”,s=4,label=“Training data”)<br>
plt.scatter(test_data,test_label, c=“g”,s=4,label=“Testing data”)<br>
if predictions is not None:<br>
plt.scatter(test_data,predictions,c=‘r’,s=4,label=“Predictions”)</p>
<p>plt.legend(prop={“size” : 14})</p>
<p>plot_predictions()<br>
error:-NameError                                 Traceback (most recent call last)<br>
 in <br>
26   plt.scatter(train_data,train_label,c=“b”,s=4,label=“Training data”)<br>
27   plt.scatter(test_data,test_label, c=“g”,s=4,label=“Testing data”)<br>
—&gt; 28 if predictions is not None:<br>
29   plt.scatter(test_data,predictions,c=‘r’,s=4,label=“Predictions”)<br>
30 ‘’‘’)</p>
<p>NameError: name ‘predictions’ is not defined</p>
<p>Can any one help me to solve this ?</p>

ptrblck: <p>Your code is not formatted properly (you can post code snippets by wrapping them into three backticks ```), but generally it should work as seen here:</p>
<pre><code class="lang-python">Training_SetofX = np.random.randn(10, 10)

def plot_predictions(train_data = Training_SetofX,
                     predictions = None):
    plt.figure(figsize = (10,7))
    if predictions is not None:
        plt.scatter(train_data, predictions, c='r', s=4, label="Predictions")

plot_predictions()
plot_predictions(predictions=np.random.randn(10, 10))
</code></pre>

Teddy_Salas: <p>it should, but i am using google co-lab, but i am keep seeing the error</p>

Teddy_Salas: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i solved the problem</p>

ptrblck: <p>Great! How did you solve it?</p>

Teddy_Salas: <p>it was the “prediction”  i put prediction in if statement, it was “predictions”  in function parameter.</p>
Partial Gradient Tracking?

Jeremy_Cowles: <p>Hey all,</p>
<p>We have a model that has large capacity, but each training step only updates a tiny subset of those parameters. The forward pass behaves the same, it only uses a small subset of the total parameter space, though the parameters are allocated in a contiguous tensor (imagine looking up the required parameters in a table based on each input).</p>
<p>Is it possible to partially track gradients on each step to reduce peak VRAM use? Or does anyone have advice on how to structure this so it scales proportionally to the actual number of parameters updated?</p>
<p>Note that the parameter tensors are not “sparse”, all elements of the tensor have values, it’s just that many values don’t change for a given step - so I don’t /think/ SparseAdam applies, but maybe I’m wrong.</p>
<p>–<br>
Jeremy</p>

ptrblck: <p>I don’t believe your use case is easily doable since gradients will be calculated for the entire parameter and not for subtensors.<br>
You could check <a href="https://discuss.pytorch.org/t/custumize-the-backpropagation-phase-of-a-neural-network-to-ignore-the-update-of-some-parameters/167557/4">this approach</a> which uses subtensors with a frozen and trainable part and might fit your use case. I’ve also posted a way to replace built-in layers with this custom module using <code>torch.fx</code> in case that’s helpful.</p>

Jeremy_Cowles: <p>Thanks! Yeah, I don’t think frozen subsets will work, because the subset is based on a hash and is a different set of scattered elements for every item in the batch</p>
Non speech audio embedding

Brecht_De_Cock: <p>Hi,<br>
I have a large dataset of machine sound waveforms (normalized from -1 to 1 in value) that i want to use in a transformer model. This means i need to get the audio in a vector representation, but i do not know how to achieve this. I assume going for speech trained models like wav2vec or music models is not the way to go.<br>
Any tips/pointers are welcome!</p>
<p>[edit]: this might be stupid, but can the audio signal be seen as a vector in and of itself? No embeddings have to be done and the waveform can be a direct input to the transformer?</p>
<p>Thanks in advance</p>
When not to use batch normalisation?

Anuj_Chopra: <p>I have a task to predict a disease from ECG of a person. An ecg in raw form is a 2d matrix ( num leads X num seconds). Sometimes the ecg reading is also dependent ( and varies slightly ) according to the machine ( or brand ) used to capture it. GE machine could have different distribution than Philips but very less difference. I have trained a 1-D conv model with residual connections for this. There is 5% performance drop on model trained on GE machine when I predict on Philips ECG.<br>
Could anyone help me with figuring out whether removing Batch Normalisation during training will help in generalising the model so that I get similar performance on other machine types as well?</p>

J_Johnson: <p>What you’re describing sounds like a perfect case for batch normalization. You’ve stated:</p>
<ul>
<li>The ECG reading can be brand/machine dependent;</li>
<li>ECG is provided in raw form from the machine without any normalization;</li>
</ul>
<p>What you <em>should</em> do is train the model with data from a wide range of machines and most definitely include batch norm layers.</p>
<p>The best corollary I can give you is how our eyesight works. When you go from a dark room to a bright room, your eyes “adjust” to the new light settings. Likewise when you go from a room with a yellow lighting to a blue lighting or full spectrum lighting, everything looks different until our eyes adjust.</p>
<p>Similarly, batchnorm layers are trained to normalize the incoming data into a range that the convolution layers are most effective in.</p>

Anuj_Chopra: <p>During training I have data from only 1 machine but I want to improve performance on other machines.<br>
A doctor when shown an ecg from any machine is able to detect the problem as he can adjust his reading according to machine. The model has performance drop.<br>
Some machines can have little higher peaks and lower troughs. Some machines can have some bias in reading. This is only difference which occurs.<br>
Do you still think I should use batch norm to train?</p>

J_Johnson: <p>If you’re only training from the data from one machine, then it might not make a difference. So I’m not really clear on what your question is, then.</p>
<p>The doctors who make those adjustments do so through trial and error. It’s a learning process. However, the outputs of a model during inference time are not going to engage in “trial and error” unless you specifically make a trainable layer and pre-program it to fine-tune after a certain number of data samples have accrued.</p>