akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can I use resnet50 PyTorch model as a Siamese network

S_M: <p>Hi all, I use contrastive loss to train resnet50 in form of a Siamese network on CIFAR-10 dataset.<br>
Siamese networks gets two images as input and the here I get two logits of the network as output1 and output2. They would be compared with each other using contrastive loss.<br>
the truth labels are 0 and 1 and I set the last linear of resnet50 to 10 neurons as my logits layer could be had any number of neurons.</p>
<blockquote>
<pre><code>net = models.resnet50(pretrained=False)
net.fc = nn.Linear(net.fc.in_features, 10)
</code></pre>
</blockquote>
<p>my contrastive loss is as below:</p>
<pre><code class="lang-auto">class ContrastiveLoss(nn.Module):
    def __init__(self, margin=1.):
        super(ContrastiveLoss, self).__init__()
        self.margin = margin
        self.eps = 1e-9

    def forward(self, output1, output2, target):
        distances = (output2 - output1).pow(2).sum(1)
        losses = 0.5 * (target.float() * distances +
                        (1 + -1 * target).float() * F.relu(self.margin - (distances + self.eps).sqrt()).pow(2))
        return losses.mean()
</code></pre>
<p>where output1 and output2 are the output of resnet50 network.<br>
Accuracy would be calculated as follows for any images. Total accuracy is a averaged accuracies of the two logits of images.</p>
<pre><code class="lang-auto">def accuracy_contrastiveLoss(y_true, y_prob):
    y_prob = np.where(y_prob.cpu().numpy() &lt;= 0.5, 0, y_prob.cpu().numpy())
    y_prob = np.where(y_prob &gt; 0.5, 1, y_prob)
    y_prob = torch.tensor(y_prob).to('cuda')
    return torch.sum(y_prob == y_true).item()
</code></pre>
<p>unfortunately my accuracy didn’t increase more than 53%. what is the problem in this scenario?<br>
any help would be appreciated.</p>
Global constants with torchscript

beakin: <p>Ok, I know that global constants defined in the base of a python file don’t work with <code>torch.jit.script()</code>, but I’ve tried 4 or 5 seemingly reasonable ways to define a collection of constants for torchscript consumption and none of them work.</p>
<p>Does anyone have a good solution for project-wide constants used in a range of functions/modules across different files short of passing around and indexing a dict?  That seems 1) inefficient as the constants can’t be compiled down as constants, 2) like bad code since it’s not obvious the dict shouldn’t be mutated.  I’ve considered just hard coding constants across the codebase, but some are used in 20+ places and <strong>could</strong> actually change in the future despite not being a parameter, creating a real maintenance headache (think physical device measurements).</p>
<p>Surely there’s a best practice way to have constants available without a lot of object passing??  Any advice would be very much appreciated.</p>

Sami_Liedes: <p>It would indeed be really nice to have some good way to do this. I’m struggling with this now. It feels like I need to rather significantly restructure my code to make it work with torch.jit.script.</p>
<p>If I for example have a global (or some other kind of) constant float or tensor, could something like this exist:</p>
<pre><code class="lang-auto">
x: torch.Tensor
y: float

@torch.jit.script
def func(n: int) -&gt; :
    x_const = torch.jit.assume_constant(x)
    return x_const + x_const[n] + torch.jit.assume_constant(y)
</code></pre>
<p>?</p>
Help me understand CUDA multiprocessing

sami: <p>Hey everyone!<br>
I’m trying to build a docker container with a small server that I can use to run stable diffusion.<br>
I’ve used the example code from banana.dev as a base and have uploaded my container to runpod.<br>
In the server, I first call a function that initialises the model so it is available as soon as the server is running:</p>
<pre><code class="lang-python">from sanic import Sanic, response
import subprocess
import app as user_src
import torch

# We do the model load-to-GPU step on server startup
# so the model object is available globally for reuse
user_src.init()

# Create the http server app
server = Sanic("my_app")

@server.route('/', methods=["POST"]) 
def inference(request):
    try:
        model_inputs = response.json.loads(request.json)
    except:
        model_inputs = request.json

    output = user_src.inference(model_inputs)

    return response.json(output)


if __name__ == '__main__':
    torch.multiprocessing.set_start_method('spawn', force=True)
    server.run(host='0.0.0.0', port=8000, workers=1)
</code></pre>
<p>the actual model is defined in the app.py in which first the model is initialised:</p>
<pre><code class="lang-python">
def init():
    global model
    HF_AUTH_TOKEN = os.getenv("HF_AUTH_TOKEN")
    
    # this will substitute the default PNDM scheduler for K-LMS  
    lms = LMSDiscreteScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear")

    model = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=lms, use_auth_token=HF_AUTH_TOKEN).to('cuda')
</code></pre>
<p>and then the inference function uses the global model variable to run inference:</p>
<pre><code class="lang-auto">def inference(model_inputs:dict) -&gt; dict:
    global model

[...]
    
    # Run the model
    with autocast('cuda'):
        images = model(
            prompt,
            width=width,
            height=height,
            num_inference_steps=num_inference_steps,
            guidance_scale=guidance_scale
        )["sample"]

[...]
</code></pre>
<p>Every time I’m trying to run inference, I’m getting a <code>RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method</code> error, even though I have a) added the call to use the spawn start method to the server, and b) am (to my understanding) not using multiprocessing. Can you help me understand how CUDA initializes under the hood and how I can fix these errors?</p>
<p>Best and thank you,</p>
<p>Sami</p>

Laoliu_Gou: <p>I have the same problem, did you solve it?</p>

sami: <p>Yes, I solved it by not using Sanic anymore. We’re now using a sns message queue from aws, which does not spawn a child process. Maybe you can try using a Flask server or something to avoid the problem <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
TypeError: conv2d() received an invalid combination of arguments - got (torch.device, Parameter, NoneType, tuple, tuple, tuple, int), but expected one of:

Arupreza: <p>Dear concern,<br>
I need help with calling the trained model and getting prediction results from it. I really appreciate any help you can provide.</p>
<pre><code class="lang-auto">def get_prediction(model, dataloader):
    with torch.no_grad():
        model = model.to(device)
        model.eval()
        prediction = np.zeros(len(dataloader.dataset))
        labels = np.zeros(len(dataloader.dataset))
        k = 0
        for images, target in dataloader:
            if device:
                images = images.type(torch.cuda.FloatTensor)
                images = images.device
            prediction[k:k+len(images)] = np.argmax(model(images).data.cpu().numpy(), axis=1)
            labels[k:k+len(images)] = target.numpy()
            k += len(images)
    return prediction, labels

pred = get_prediction(model, valid_loader)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf8c27aa37d67204556306c551b59d5f1958f03d.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf8c27aa37d67204556306c551b59d5f1958f03d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf8c27aa37d67204556306c551b59d5f1958f03d.png" alt="image" data-base62-sha1="tC37fAGf7aWckhexcroHdAm1E05" width="690" height="113" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">987×162 10.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>You are overriding the <code>images</code> tensor with its <code>device</code> attribute in:</p>
<pre><code class="lang-python">images = images.device
</code></pre>
<p>and try to pass it to the model.<br>
Pass the tensors instead and it should work.</p>

Arupreza: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Partial derivative (torch.autograd.grad) within torch.no_grad()

n_kotel: <p>Hello,</p>
<p>I have an HMC (Hamiltonian Monte Carlo) sampler, parametrised by neural networks.</p>
<p>After training procedure when performing sampling, I do not need to compute a backward graph, and that is why I would like to use torch.no_grad() upon my model to speed up the procedure.</p>
<p>But at the same time to perform a leapfrog step, I have to compute the partial derivative of log posterior, which only depends on the current coordinate (I always can manually set requires_grad=True to compute this derivative, but not within torch.no_grad()).</p>
<p>Is there a clever way, how one could use torch.autograd.grad within torch.no_grad()?</p>
<p>Thank you!</p>

JuanFMontesinos: <p>You can use torch.grad() guardian the same way you use torch no grad to compute grads in that specific part.</p>

ludwigwinkler: <p>Do you mean the decorator  <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_grad_enabled" rel="nofollow noopener">https://pytorch.org/docs/stable/autograd.html#torch.autograd.set_grad_enabled</a><br>
?</p>

n_kotel: <p>Thank you, it works!</p>

JuanFMontesinos: <p>Ups I though it was torch.grad() but yeh, enable_grad() or set_grad_enabled. Any of them should work.</p>

ximing: <p>Hi May I ask how you makes it work?</p>
Profiler time do not match NVIDIA Nsight Computer

lk137095576: <p>I profiled the fc layer with torch.profiler, got kernel time <strong>1.9 ms</strong>  ,</p>
<pre><code class="lang-auto">with profile(activities=[ProfilerActivity.CUDA, ProfilerActivity.CPU], record_shapes = True, profile_memory = True, with_modules=True, )
    model(data)
prof.export_chrome_trace("output.json")
</code></pre>
<p>but I got kernel time  <strong>2.2ms</strong> from nsight computer.<br>
there is 10% miss, where is the error?<br>
It is important for me</p>
Model's parameters for calculating the weight are not updated

malioboro: <p>Hi all, please help me,<br>
I want to make a layer that its weight value (and the bias) is based on the other frozen weight. So, let’s say I have a frozen weight (FW), then my current model layer will have weight FW + D, where D is the trainable parameter. I hope, later, when I train the model, the only parameter that gets updated is D.</p>
<p>I made this simple code for illustration:</p>
<pre><code class="lang-auto">frozen= nn.Linear(100,10)
frozen.weight.requires_grad = False
frozen.bias.requires_grad = False

class Net(nn.Module):
    
    def __init__(self):
        super(Net, self).__init__()
        self.fc = nn.Linear(100,10)
        self.dw = nn.Parameter(torch.tensor(1.0, requires_grad=True))
        self.db = nn.Parameter(torch.tensor(1.0, requires_grad=True))

    def forward(self, x):
        self.fc.weight = nn.Parameter(torch.add(frozen.weight, self.dw))
        self.fc.bias = nn.Parameter(torch.add(frozen.bias, self.db))
        return torch.sigmoid(self.fc(x))
    
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.rand(100)
y = torch.tensor([0]*9+[1], dtype=torch.float32)

for _ in range(10):
    out = model(x)
    loss = criterion(out, y)
    print(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>But when I run that code, the model doesn’t train, the <code>self.dw</code> and <code>self.db</code> doesn’t change. I am not sure whether my concept is wrong, so it’s not possible to train D, or I made a mistake in the implementation.</p>
<p><strong>Update</strong>:<br>
I tried to implement using nn.utils.parameterize, but it still doesn’t work</p>
<pre><code class="lang-auto">frozen = nn.Linear(100,10)
frozen.weight.requires_grad = False
frozen.bias.requires_grad = False

class Adder(nn.Module):
    def __init__(self, delta, frozen):
        super().__init__()
        self.delta = nn.Parameter(torch.tensor(delta, requires_grad=True))
        self.frozen=frozen
    def forward(self, x):
        return torch.add(self.frozen, self.delta)    

class Net(nn.Module):
    def __init__(self):    
        super(Net, self).__init__()
        self.fc = nn.Linear(100,10)

    def forward(self, x):
        nn.utils.parametrize.register_parametrization(self.fc, "weight",  Adder(1.0, frozen.weight))
        nn.utils.parametrize.register_parametrization(self.fc, "bias", Adder(1.0, frozen.bias))

        return torch.sigmoid(self.fc(x))
    
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.rand(100)
y = torch.tensor([0]*9+[1], dtype=torch.float32)

for _ in range(10):
    out = model(x)
    loss = criterion(out, y)
    print(loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>Thank you for any responses.</p>
ValueError: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([13456, 1])) is deprecated. Please ensure they have the same size

easonoob: <p>My code: <a href="https://colab.research.google.com/drive/1qjfy2OsHYewhHDej-W83CMNercB7o7r8?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Google Colab</a></p>
<p>the error: ValueError: Using a target size (torch.Size([16])) that is different to the input size (torch.Size([13456, 1])) is deprecated. Please ensure they have the same size.</p>
<p>I wonder where does the 13456 come from?</p>
<p>the dataset consists of 2 folders: 0 and 1, and in each of these two folders, there’re about 2500 512*512 images and a json file for each image.</p>
<p>the code was from the pytorch gan tutorial, i just changed the dataset.</p>
I heard the Pytorch team is adding the flash-attention for Transformer

JonathanSum: <p>If so, when are we going to have it?<br>
I have to ask the team because flash-attention is not a model implementation. It is the implementation of matrix multiplication.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/307335cd0dc5689596d5aebe8863fe1bed2f6314.png" alt="image" data-base62-sha1="6UBKP1YIGfez89UHg3gEk9rH6Ek" width="599" height="375"><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fbc6d05e3eb1d3d19eefb40989123af4f024c12.png" data-download-href="https://discuss.pytorch.org/uploads/default/5fbc6d05e3eb1d3d19eefb40989123af4f024c12" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fbc6d05e3eb1d3d19eefb40989123af4f024c12_2_690x225.png" alt="image" data-base62-sha1="dEV3JDzFNjheBR4JUYC4VMlkMGS" width="690" height="225" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/f/5fbc6d05e3eb1d3d19eefb40989123af4f024c12_2_690x225.png, https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fbc6d05e3eb1d3d19eefb40989123af4f024c12.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/f/5fbc6d05e3eb1d3d19eefb40989123af4f024c12.png 2x" data-dominant-color="F1F0F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">920×301 72.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Error calling a nn network

sojohan: <p>Hi Experts I have this very simple resnet network but I get an error at the end. Do you have hints ?</p>
<p>This is the first network:</p>
<pre><code class="lang-auto">class mlp(nn.Module):
    def __init__(self,nfeatures,nout):
        super(mlp, self).__init__()
        self.fc1 = nn.Linear(nfeatures,nout)
        self.tanh1=nn.Tanh()
    def forward(self, x):        
        output = self.fc1(x)
        input=self.tanh1(output)
        return outputtype or paste code here
</code></pre>
<p>And is called by this code:</p>
<pre><code class="lang-auto">tclass resnet(nn.Module):
    def __init__(self,nfea,out,depth=3):
        super().__init__()
        net=mlp(nfea,out)
        self.nfea=nfea
        self.out=out
        self.dep=depth
        self.resblock = nn.Sequential(*(self.dep*[net]))
        #print('hej')
        def forward(self, x):
            out=self.resblock(x)+x
            return outype or paste code here
</code></pre>
<p>I can do:</p>
<p>model=resnet(10,2,1).to(device)</p>
<p>But when I do:<br>
inputs=torch.linspace(-2.0,2.0,10)</p>
<p>model(inputs)</p>
<p>I get cryptic error : NotImplementedError:</p>
<p>Any hints?</p>

ptrblck: <p>The indentation of the <code>forward</code> method in <code>resnet</code> is wrong, so decrease the indent (move it to the left) and it should work.</p>

sojohan: <p>Yes of course. Thanks. <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20"></p>
KL divergence returns NaN on mps

cafaxo: <p>I am running into the following issue:</p>
<pre><code class="lang-auto">Python 3.10.8 (main, Oct 13 2022, 09:48:40) [Clang 14.0.0 (clang-1400.0.29.102)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; ref = torch.Tensor([0., 0., 1.]);
&gt;&gt;&gt; pred = torch.Tensor([-1.09861, -1.09861, -1.09861]);
&gt;&gt;&gt; torch.nn.functional.kl_div(pred, ref, reduction='batchmean')
tensor(0.3662)
&gt;&gt;&gt; refmps = ref.to('mps');
&gt;&gt;&gt; predmps = pred.to('mps');
&gt;&gt;&gt; torch.nn.functional.kl_div(predmps, refmps, reduction='batchmean').cpu()
tensor(nan)
</code></pre>
<p>I am using pytorch 1.13.0. Am I doing something wrong here or is this an issue I should report to github?</p>
Tensor.to(device)

cat03: <pre><code class="lang-python">import torch

x = torch.randn([100, 3, 28, 28])
device = torch.device("cuda:0")
x = x.to(device)
</code></pre>
<p>I wanna trace the procedure tensor.to(device)，research the function calls and how the cuda memory is allocated and tensor moved. But there are so many python tricks within the torch source code. I wanna ask where the Tensor.to() is defined？Thanks</p>

ptrblck: <p>The <code>.to()</code> operation should eventually dispatch to any of the <code>r.copy_(self, non_blocking)</code> calls in <a href="https://github.com/pytorch/pytorch/blob/8dc3353b0b1c12f64ba790c7be85cfbc99448cb4/aten/src/ATen/native/TensorConversions.cpp#L223"><code>_to_copy</code></a> and then into <a href="https://github.com/pytorch/pytorch/blob/08820cb030927200c92733436f454f5fcf19c6bd/aten/src/ATen/native/cuda/Copy.cu#L160"><code>copy_kernel_cuda</code></a>.</p>

cat03: <p>Thanks for your help.</p>
Does jit support TORCHVISION.OPS.ROI_ALIGN?

Soonmyun_Jang: <p>After making a pt file using torch.jit.trace(), I tried to load it in QT with C++ but I met the problem:</p>
<p>Could not find any similar ops to torchvision::roi_align. This op may not exist or may not be currently supported in TorchScript.</p>
<p>Does not torch support roi_align()?</p>
<p>Please help me to solve it if you have any idea.</p>
Matrix Multiplication Along Axis

Thomas_Ahle: <p>I have a bunch of matrices M1, M2, …, Mk in a tensor of shape (k, d, d). I want to compute the matrix product M1 @ M2 @ … @ Mk. This would give an output tensor of shape (d, d). Is there a fast way to do this in PyTorch?</p>
<p>I looked at some questions that claim to be about this <a href="https://discuss.pytorch.org/t//100867" class="inline-onebox">How do do matrix multiplication (matmal) along certain axis?</a> and <a href="https://discuss.pytorch.org/t//138409" class="inline-onebox">Matrix multiplication along specific dimension</a> , but they seem to be concerned with normal tensor contraction, as can be done with einsum. I don’t think einsum can solve my problem. Or am I missing something?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/thomas_ahle">@Thomas_Ahle</a>,</p>
<p>You can use <code>torch.linalg.multi_dot</code>, which will compute the optimal ordering of your <code>k</code> number of matrices. Although, you’ll need to pass your <code>(k, d, ,d)</code> tensor as a list of matrices instead. That can be done through list comprehension.</p>
<pre><code class="lang-auto">import torch
matrices = torch.randn(4, 6, 6)
torch.linalg.multi_dot([m for m in matrices])
"""
returns 
tensor([[-20.4766, -18.9575,   2.6211,  -9.0951,   7.8230,   6.8106],
        [ 13.0261,   5.3627, -18.5586,  -5.2984, -24.7720,   0.4994],
        [  6.4090,  14.7333,   2.8740,   3.7405,   5.7821,  -7.2868],
        [ -2.2964,   0.6236,   8.3580,  -0.3444,   5.8850,  -6.5138],
        [-14.0015,  -8.9284,  15.0120,   0.9839,  18.0842,  -0.1127],
        [-10.8341,  -0.6622,   9.4417,   2.6874,  19.8331,   1.5707]])
"""
</code></pre>

Thomas_Ahle: <p>Hi <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a>, the ordering doesn’t matter so much for me, since the matrices are all square.</p>
<p>What I want to achieve is avoiding the python loop.</p>
<p>I tried timing the simple</p>
<pre><code class="lang-auto">def mymul(mats):
  res = mats[0]
  for mat in mats[1:]:
    res @= mat
  return res
</code></pre>
<p>against <code>multi_dot</code>:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; matrices = torch.randn(10**3, 2, 2)
&gt;&gt;&gt; %timeit mymul(matrices)
3.72 ms ± 24.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
&gt;&gt;&gt; %timeit torch.linalg.multi_dot(list(matrices))
585 ms ± 14.2 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
</code></pre>
<p>So <code>torch.linalg.multi_dot(matrices)</code> is more than 100x slower than the naive python version. I also tried 10^4 matrices, but then <code>multi_dot</code> just crashed.</p>
Concat Shuffled for DataPipes

Thomas_Ahle: <p>Say I have a number of files, and I want to load all the data and shuffle it. The data would be too big for my memory, but I could shuffle the files individually, and then merge them using an “external shuffle”.</p>
<p>I imagine it would look something like <code>FileLister(".").open_files().map(my_parser).in_batch_shuffle().concat_shuffled()</code>.</p>
<p>So if <code>FileLister(".").open_files().map(my_parser)</code> gives <code>[[1,2,3],[4,5,6]]</code> then after <code>in_batch_shuffle()</code> we may have <code>[[3,1,2],[4,6,5]]</code>, and after <code>concat_shuffled()</code> we’ll have <code>[4, 6, 3, 1, 5, 2]</code>.</p>
<p>Right now I can just do</p>
<p><code>FileLister(".").open_files().flatmap(my_parser).shuffle()</code></p>
<p>but that requires a fixed buffer size, so it won’t be a true shufle of my data.</p>
<p>Btw, an external shuffle is pretty easy to implement doing something like this:</p>
<pre><code class="lang-auto">def cshuf(x, y):
  while x or y:
    px = len(x)/(len(x) + len(y))
    if random.random() &lt; px:
      yield x.pop(0)
    else:
      yield y.pop(0)
</code></pre>
<p>Is there already a way to do this with DataPipes? Or is there a way I can implement it?</p>
Kernel dies when I try to use torch on jupyter notebook

helga_thorey: <p>Please help me I have tried everything. I have tried to downgrade torch. This happens every time I try to run my code:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a48a13637501a4894b49604f34bff6665fe3d81.png" data-download-href="https://discuss.pytorch.org/uploads/default/4a48a13637501a4894b49604f34bff6665fe3d81" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a48a13637501a4894b49604f34bff6665fe3d81_2_690x172.png" alt="image" data-base62-sha1="aB8Wmm3LYLQpb65rnIF6g7X5ieJ" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a48a13637501a4894b49604f34bff6665fe3d81_2_690x172.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a48a13637501a4894b49604f34bff6665fe3d81_2_1035x258.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a48a13637501a4894b49604f34bff6665fe3d81_2_1380x344.png 2x" data-dominant-color="B7B8B8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2090×522 48.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

vdw: <p>You probably want to clarify what your code is doing. I assume the kernel does not idea from simply running <code>import torch</code></p>
<p>When one of my kernels dies, then it’s usually always because of running out memory. Are you trying to load huge chunks of date all at once?</p>

helga_thorey: <p>It dies just by running these imports:</p>
<p>from skimage import color, io, measure, img_as_ubyte, draw<br>
from skimage.measure import profile_line<br>
from skimage.transform import rescale, resize<br>
import matplotlib.pyplot as plt<br>
import numpy as np<br>
import pydicom as dicom<br>
from os import listdir<br>
from os.path import isfile, join<br>
from skimage.filters import prewitt_h<br>
from skimage.filters import prewitt_v<br>
from skimage.filters import prewitt<br>
from skimage.filters import median<br>
from skimage.filters import gaussian<br>
from torch.utils.data import Dataset<br>
import nltk<br>
import torch<br>
import torch.nn as nn<br>
<span class="hashtag">#import</span> torchvision.transforms.functional as TF</p>

vdw: <p>Did you try each imports individually to see which fails, or do all fail. If the latter, is would seem to be a bigger issue with your setup/system.</p>
LSTM variable sequence length

Erfan: <p>hello,<br>
i want to use LSTM for below data structure. i want to predict next rating of review.<br>
first: business and rating (1-5 stars)<br>
Business_1 = [1, 3, 4, 5, 2, 4, 1, 1, 4, 5, 2, 1] =&gt; length = 12<br>
Business_2 = [3, 3, 2, 2, 1, 5, 4]                     =&gt; length = 7<br>
Business_3 = [1, 3]                                         =&gt; length = 2</p>
<p>how I can use nn.LSTM when the length of data is different? if I set L= 3 for Business_3 is not ok!!</p>
<p>i have 2 problems.</p>
<ol>
<li>set L for LSTM</li>
<li>different length of data for each business. i want to use one LSTM model for all business.</li>
</ol>
<p>my second project is like it but with emotion.( 5 categorial emotion = [happy, sad, neutral , angry, fear]<br>
Business_1 =<br>
{review1: happy:0.4, sad:0.1, fear:0.0, angry:0.2, neutral: 0.3<br>
review2: happy:0.0, sad:0.1, fear:0.0, angry:0.1, neutral: 0.7<br>
review3: happy:0.6, sad:0.0, fear:0.0, angry:0.2, neutral: 0.2<br>
}<br>
Business_2=<br>
{review1: happy:0.4, sad:0.1, fear:0.0, angry:0.2, neutral: 0.3<br>
review2: happy:0.3, sad:0.5, fear:0.1, angry:0.1, neutral: 0.0<br>
review3: happy:0.4, sad:0.0, fear:0.1, angry:0.1, neutral: 0.4<br>
}</p>
<p>i want to predict the type of emotion and the amount of it for next review.</p>
<p>thank you very much for your response from now.</p>

vdw: <p>You can check out <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284">this post</a>. You can either pad all shorter sequence in batch, or ensure that each batch contains only sequences of the same length – or a combination of both approaches.</p>
Encoder-Decoder model for variable length waveform

kkchan: <p>Hi There,<br>
I want to build a network which inputs a variable length waveform and then outputs exactly the same length waveform as input. Is there any recommendations to build such network. I try to use conv1d and transposed1d to build the network, but during the downsample stage some layers would round the size of last axis to an int, and cause the final output waveform smaller than the input.</p>
<p>Thanks</p>
Set a specific layer/operator to use non-deterministic mode

pepper8362: <p>Hi,</p>
<p>I try to use <code>torch.use_deterministic_algorithms(True)</code>, but some layers in my network does not support determinism. Is there a way to set those layers that do not support determinism to work in non-deterministic mode, while the other layers in deterministic mode?</p>
<p>Thanks.</p>

ptrblck: <p>No, I don’t think it’s possible to pick determinism for each module and <code>use_deterministic_algorithms</code> enables determinism for all used algorithms (or raises a warning/error if that’s not possible).</p>
How to convert tensorflow's BatchNormalization to pytorch?

YuhskeHujisaki: <pre><code class="lang-auto">BatchNormalization(axis=3, epsilon=1e-05, momentum=0.1)
</code></pre>
<p>I want to convert this tensoflow code to pytorch, but pytorch does not have axis, how should I convert it?<br>
Integer, the axis to be normalized (usually the feature axis). For example, set axis=1 in BatchNormalization after a Conv2D layer with data_format=“channels_first”. However, pytorch does not allow you to specify an axis.</p>
What is kernel_constraint=max_norm(2., axis=(0, 1, 2))?

YuhskeHujisaki: <pre><code class="lang-auto">Conv2D(self.filter_1, (1, 64), activation='elu', padding="same", 
                                kernel_constraint=max_norm(2., axis=(0, 1, 2)))(encoder_input)
</code></pre>
<p>This is tensoflow code.<br>
What is kernel_constraint=max_norm(2., axis=(0, 1, 2))?<br>
I want to convert this code to pytorch code, but I don’t know how to express the kernel_constraint=max_norm(2., axis=(0, 1, 2)) part. How should I convert it?</p>
PyTorch 1.13: RuntimeError: tensors used as indices must be long, byte or bool tensors

Sergei_Vorobev: <p>I have the following code that works on PyTorch 1.11 and fails on 1.13</p>
<pre><code class="lang-python">import torch

print(torch.__version__)
model = torch.jit.load("foo/ranker.pt")
model = model.eval()
x = torch.jit.load("foo/tensor_all.pt")
inputs = list(x.parameters())
for i in inputs:
    i.requires_grad = False
model.forward(*inputs)
print("ALL GOOD")
</code></pre>
<p>The error that it fails with is</p>
<pre><code class="lang-auto">worker_exedir/cruise/mlp/prediction/pytorch/modules/utils/utils.py", line 81, in gather_from_map

    map_idx = torch.stack([batch_indices, y_indices, x_indices], dim=-1)
    map_features = map[map_idx[..., 0], map_idx[..., 1], map_idx[..., 2], :]
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    return map_features
RuntimeError: tensors used as indices must be long, byte or bool tensors
</code></pre>
<p>We had the same error in the eager mode and we fixed it.<br>
Is there a way to fix it in a TorchScript without re-training the model?</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/sergei_vorobev">@Sergei_Vorobev</a>,</p>
<aside class="quote no-group" data-username="Sergei_Vorobev" data-post="1" data-topic="166321">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sergei_vorobev/48/54955_2.png" class="avatar"> Sergei_Vorobev:</div>
<blockquote>
<pre><code class="lang-auto">    map_idx = torch.stack([batch_indices, y_indices, x_indices], dim=-1)
</code></pre>
</blockquote>
</aside>
<p>What type are <code>batch_indices, y_indices, x_indices</code>? I assume they are <code>torch.float32</code>?</p>
<p>You should be able to fix this via adding <code>map_idx = map_idx.to(torch.long)</code> between the <code>map_idx = ... </code> line and <code>map_features = ... </code> lines.</p>

Sergei_Vorobev: <p>Thank you <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> !<br>
I understand how to do this in the python code (eager mode).<br>
Is there a way to do this with existing jit model (from .pt) file?<br>
I’d like to preserve the weights, but update the code…</p>

Sergei_Vorobev: <p>UPD: I patched the TorchScripted model with <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> 's suggestion directly. You can do that by unzipping the .pt file and finding the corresponding file (code is structured a little differently but it was easy enough to figure out the mapping in my case). Here is a helpful resource on the TorchScript serialization <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/jit/docs/serialization.md" class="inline-onebox" rel="noopener nofollow ugc">pytorch/serialization.md at master · pytorch/pytorch · GitHub</a></p>