akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Libtorch torch::stft error

akurmustafa: <p>I have saved a model as torch script, to calculate mel spectrograms. Saving the model, and using the model at python gives no error and works as expected. However when I try to use model in c++ with libtorch I have received the following error, which is unintuitive for me. I would be happy, if one explains what is wrong and how to fix it. Thanks<br>
RuntimeError: stft(CUDAFloatType[3708000, 1], n_fft=400, hop_length=80, win_length=400, window=CUDAFloatType{[400]}, normalized=0, onesided=1, return_complex=1) : expected 0 &lt; n_fft &lt; 1, but got n_fft=400<br>
]</p>

tom: <p>It would appear that the input might not have the right shape (should it be <code>1x3708000</code> or even <code>1x1x3708000</code> or something like this? If so, you need to use torch::Tensor::permute or somesuch).<br>
For me, making a habit of checking the input shapes and possible a small portion of the tensor (say slicing the first 5 elements in each dimension) has reduced a great deal of confusion with the outputs when moving between Python and C++.</p>
<p>Best regards</p>
<p>Thomas</p>

akurmustafa: <p>Thanks this indeed is the problem. Input was not in expected shape.</p>
Model parameters across GPUs are different with DDP

yongjun_Hong: <p>Hi there!</p>
<p>I am using pytorch DDP to use multi-gpus and it seems to work well until I find out the model parameters across GPUs are different. (It should be same if all-reduce operation works properly)</p>
<p>I am debugging now but still not getting it.</p>
<p>My model wrapped with DDP looks like (doesn’t include “forward” function explicitly)</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self, ...):
         ## Create other modules which have forward function
         sub_model_a = SubModel(...)
         sub_model_b = SubModel(...)

    def func_a(self, image):
         ## Do something with sub_model_a
         sub_model_a(image)
    def func_b(self, image):
         ## Do something with sub_model_b
         sub_model_b(image)
</code></pre>
<p>So my questions are</p>
<ol>
<li>Target module wrapped by DDP should have “forward” pass explicitly ?</li>
<li>If not, how can I check all-reduce function works properly ? Now, I explictly check model parameter value</li>
<li>Since model attributes are wrapped to .module after DDP, I use “func_a” and “func_b” method of MyModel class as follows</li>
</ol>
<pre><code class="lang-auto">mymodel = DDP(MyModel, device_ids=[rank])
mymodel = mymodel.module

for epoch in range(num_epochs):
    ## Just example
    loss = mymodel.func_a()
    loss.backward()
</code></pre>
<p>This pattern would be problematic with DDP ??</p>
<p>Thank you in advance.</p>

ptrblck: <p><code>DDP</code> broadcasts the <code>state_dict</code> during its construction as described in the <a href="https://pytorch.org/docs/stable/notes/ddp.html#internal-design">internal design</a>.<br>
If I understand your use case correctly, you are bypassing the <code>DDP</code> model by manually calling into the internal <code>.module</code>? If so, what’s your use case that would need this approach?</p>

yongjun_Hong: <p>Thanks for an answer.</p>
<p>Yes, you are right, I was actually bypassing the DDP model by manually calling “.module” of DDP wrapped model. I think that’s why model parameters across GPUs were different.</p>
<p>Now I created explict forward function in MyModel class, and use it instead of “func_a, func_b”, model parameters across GPUs were same. I think that DDP wrapped model trigger synchronization with forward pass</p>
Dataloader: more num_workers do not reduce runtime?

pineapple_pine: <p>Hi everyone! I am using dataloader for NN training on 1 GPU. When I increased the num_workers to 2, 4 and 8, runtime does not reduce. Can someone help to explain why it is so and how to improve runtime?</p>
<p>(Other parameters setting: Shuffle= True, pin_memory = True)</p>
<p>Thank you!</p>

tom: <p>Hi,</p>
<p>the best way depends on the data and what you are doing with it. This could mean that your CPU is saturated, that the bottleneck isn’t the CPU (but e.g. storage) or something else.</p>
<p>Some general ideas:</p>
<ul>
<li>do on-the-fly processing (e.g. augmentation) on the GPU as much as you can, i.e. not in the dataloader,</li>
<li>storage can be a huge bottleneck, e.g. don’t load huge images only to scale them down, but instead do a preprocessing step in advance where you scale down to a “reasonable size” (i.e. doesn’t need to be the final size, but I’ve seen things being slow because XX Megapixel images were loaded only to then immediately rescale + crop to 227x227,</li>
<li>if you adjust your pipeline, the Thomas rule of thumb is <em>it isn’t optimization unless you measure before and after</em>. Things like <code>pin_memory</code> are often advocated, but people sometimes find it helps and sometimes now.</li>
</ul>
<p>Best regards</p>
<p>Thomas</p>

pineapple_pine: <p>Thank you for the quick response Tom! I’ve just figured out the problem in my last reply here  <a href="https://discuss.pytorch.org/t/dataloader-num-workers-relate-to-gpu-memory/136205/6">Dataloader num_workers relate to gpu memory? - Memory Format - PyTorch Forums</a></p>
Dataloader num_workers relate to gpu memory?

dlfma_tjd: <p>If I increase num_workers from 0 to 4, then could be cuda out of memory problem solved? RAM is enough but memory is exploded…help me please</p>

ptrblck: <p>No, increasing <code>num_workers</code> in the <code>DataLoader</code> would use multiprocessing to load the data from the <code>Dataset</code> and would not avoid an out of memory on the GPU.<br>
To solve the latter you would have to reduce the memory usage by e.g. reducing the batch size or by using e.g. <code>torch.utils.checkpoint</code>.</p>

pineapple_pine: <p>Hi, ptrblck! Do you know why increasing num_workers to 2, 4 or 8 wouldn’t decrease the runtime?</p>

ptrblck: <p>You would have to profile your code and check how long the data loading takes. E.g. your overall data loading might be faster than the model training iteration and thus the time to load each batch could already be hidden. In such a case speeding up the data loading would of course not yield any performance improvement. On the other hand you could see a data loading bottleneck, but your system cannot speed it up further e.g. due to the limited read speeds of your SSD etc.<br>
Generally, I would recommend to profile the code to see where the bottlenecks are and then try to optimize it.</p>
<p>EDIT: also explained in <a href="https://discuss.pytorch.org/t/dataloader-more-num-workers-do-not-reduce-runtime/140910/2">this answer</a> from your double post.</p>

pineapple_pine: <p>Thank you so much!! I will try your suggestion.</p>

pineapple_pine: <p>May I ask how to test the batch loading speed &amp; the model training iteration speed?</p>
<p>Also, if I set num_workers = multiprocessing.cpu_count() which maximize the usage of cpu, and still does not improve runtime, does that mean there’s no way to improve runtime?</p>
<p>New to Pytorch for these silly question lol</p>

pineapple_pine: <p>I’ve figured it out. Actually setting num_workers = 0 will speed up the code because the loading time itself is short, but creating new threads actually takes much longer. Therefore, with more num_workers &amp; even using multiprocessing.cpu_count() will slow down the runs.</p>
Why NN doesnot learn when number of output nodes is larger than input nodes?

aknurkarabay: <p>Hello! I am working on Neural Network for a regression problem. I have 4 inputs and 7 outputs. But it doesn’t learn at all anything. I am applying minmax normalization for input and output (0 to 1). Optimizer: Adam, learning rate = 0.0005. Do you have any suggestions, please? Thanks in advance!<br>
This is the architecture I am using: class Net(nn.Module):<br>
def <strong>init</strong>(self, n_in, n_out):<br>
super(Net, self).<strong>init</strong>()<br>
self.fc1 = nn.Linear(n_in, 32)<br>
self.fc2 = nn.Linear(32, 64)<br>
self.fc3 = nn.Linear(64, 128)<br>
self.fc4 = nn.Linear(128, 64)<br>
self.fc5 = nn.Linear(64, 32)<br>
self.fc6 = nn.Linear(32, n_out)</p>
<pre><code>def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    x = F.relu(self.fc4(x))
    x = F.relu(self.fc5(x))
    x = self.fc6(x)
    return x</code></pre>

ptrblck: <p>Your model seems to be able to overfit random data as seen here:</p>
<pre><code class="lang-python">model = Net(4, 7)
x = torch.randn(64, 4)
target = torch.randn(64, 7)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()


for epoch in range(1000):
    optimizer.zero_grad()
    output = model(x)
    loss = criterion(output, target)
    loss.backward()
    optimizer.step()
    print("epoch {}, loss {}".format(epoch, loss.item()))
</code></pre>
<p>so you could check if changing some hyperparameters might help in your case.</p>
Torchaudio: how to frame and append long wav file?

jaden: <p>torchaudio: how to frame and append long wav files using different durations?</p>
<p>e.g., I have 40G of 100,000 wav files (5 labels) with different length.</p>
<ol>
<li>I wish to do 2, 3 and 4 durations of frames among these labels. Append the frames (I do not want miss the file)</li>
<li>How to append them without save to my disk &gt;&gt; 100,000 files?</li>
</ol>

JamesWright: <p>Hey, if by framing you mean slicing audio into series of optionally overlapping frames, see <a href="https://librosa.org/doc/main/generated/librosa.util.frame.html" rel="noopener nofollow ugc">librosa.util.frame()</a> or <a href="https://pysoundfile.readthedocs.io/en/latest/#block-processing" rel="noopener nofollow ugc">soundfile.blocks()</a>.</p>

niba_cam: <p>Thanks <a class="mention" href="/u/jameswright">@JamesWright</a>. This sounds very clear to me now. I will try to use soundfile.blocks with torchaudio.</p>
How to share a cache among multiple subprocesses when using PyTorch DDP training

cindybrain: <p>I want to share a cache among multiple processes in the same node when using ddp training. Supposed that there are N nodes, we need create N caches separately, and these subprocesses in the same node share one of the caches.<br>
If necessary, the processes on one node can access the data in the cache of other nodes.</p>
<p>I found a potential solution is to use shared memory with torch.multiprocessing, which was described in <a href="https://discuss.pytorch.org/t/how-to-share-data-among-dataloader-processes-to-save-memory/108772" class="inline-onebox">How to share data among DataLoader processes to save memory</a>.<br>
However, it is not convenient if I want to training on multiple nodes, thus I choose the method <code>torch.distributed.launch</code> rather than <code>mp.spawn</code> to init DDP training.<br>
The question is how can I share a cache among multiple subprocesses in the same node,  when using the method <code>torch.distributed.init_process_group</code> rather than the method of multiprocessing.</p>
<p>Has someone ever encounter the same problem?</p>

mrshenli: <aside class="quote no-group" data-username="cindybrain" data-post="1" data-topic="139969">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc348/48.png" class="avatar"> cindybrain:</div>
<blockquote>
<p>The question is how can I share a cache among multiple subprocesses in the same node, when using the method <code>torch.distributed.init_process_group</code> rather than the method of multiprocessing.</p>
</blockquote>
</aside>
<p>Hey <a class="mention" href="/u/cindybrain">@cindybrain</a>, there is an ongoing discussion on this topic. See: <a href="https://github.com/pytorch/pytorch/issues/64932" class="inline-onebox" rel="noopener nofollow ugc">[RFC] TorchStore - A Shared-Memory Tensor Store · Issue #64932 · pytorch/pytorch · GitHub</a></p>
<p>Does the proposed solution address your use case?</p>
<p>cc <a class="mention" href="/u/cbalioglu">@cbalioglu</a></p>

cindybrain: <p>Thanks for your reply!<br>
I think it is a great proposal, and it can solve my question! Besides, I wonder why it only chooses to support tensor type in the share-memory?  Can it be extended to share other data type?</p>

cbalioglu: <p>Hi <a class="mention" href="/u/cindybrain">@cindybrain</a>, happy to hear that!</p>
<p>In fact the underlying API will allow you to store arbitrary blobs in shared memory. We deliberately limited the scope in the RFC to have feedback on the proposal’s core functionality.</p>
Elementwise Multiplication Across Batches

J_Johnson: <p>Suppose I have a tensor of images as follows:</p>
<p><code>A=torch.rand(10, 3, 28, 28)</code></p>
<p>And suppose I have another tensor that I want to be multiplied elementwise by all images and layers of A as follows:</p>
<p><code>B=torch.rand(28, 28)</code></p>
<p>Is there an easier/faster/more efficient way to do this, <strong>instead</strong> of the following?:</p>
<pre><code class="lang-auto">B=torch.cat([torch.cat([B.unsqueeze(0)]*A.size()[-3]).unsqueeze(0)]*A.size()[-4])
C=A*B
</code></pre>
<p>TIA</p>

KFrank: <p>Hi J!</p>
<aside class="quote no-group" data-username="J_Johnson" data-post="1" data-topic="140881" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>Is there an easier/faster/more efficient way to do this</p>
</blockquote>
</aside>
<p>You may use the expression for element-wise multiplication, <code>C = A * B</code>,<br>
and pytorch will use <a href="https://pytorch.org/docs/1.10.1/notes/broadcasting.html" rel="noopener nofollow ugc">broadcasting</a> to multiply all of the images and channels<br>
in <code>A</code> by <code>B</code>.</p>
<p>Note that broadcasting treats “missing” leading dimensions as if they were<br>
singleton dimensions, so <code>C = A * B.unsqueeze (0).unsqueeze (0)</code><br>
would be equivalent (but, to be clear, there is no reason, even stylistic,<br>
to use <code>unsqueeze()</code> in this case).</p>
<p>Best.</p>
<p>K. Frank</p>

J_Johnson: <p>I see. Not sure why I was getting an error before, but I’m not seeing it now. Thank you.</p>
Weighted binary cross entropy loss using pos_weight

ailneni_rao_19210068: <p>Hi,<br>
I have a unbalanced dataset, so i tried to use pos_weight in BCEwithlogit loss:</p>
<blockquote>
<p>torch.nn.BCEWithLogitsLoss(pos_weight=weights)(outputs,targets)<br>
But I observed the loss is fluctuating very badly and results are also bad. shouldn’t they atleast be on par with the results i got without using class weights. Can anyone tell why this is happening and suggest another way for using class weights.</p>
</blockquote>

mMagmer: <p>What are the metrics that you’re monitoring?<br>
If you set weights equal to <code>\#mojority/\#minority</code>, then you should be checking average recall.</p>

ailneni_rao_19210068: <aside class="quote no-group" data-username="mMagmer" data-post="2" data-topic="140878">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mmagmer/48/43283_2.png" class="avatar"> mMagmer:</div>
<blockquote>
<p>mojority</p>
</blockquote>
</aside>
<p>I’m using F1 score (both micro and macro)</p>

mMagmer: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/5/850975e21242443ad885d6641264fe8e573dcb7a.png" alt="image" data-base62-sha1="iYTLTHP6Vt3VUSCNg6jQLkVHN9g" width="410" height="35"></p>
<p>F1 score is calculated from the precision and recall .<br>
by change p_c you’re changing these two metric in opposite way.<br>
and in practice you’re gaining some recall percentage for way worse precision.</p>
Two Branch Network and Masking one Class

shakeel608: <p>Suppose I have a network as mentioned below. I have an encoder plus two branches.<br>
One branch is predicting if the sample is music or not music. if prediction is music then follow the isMusicNet, otherwise if prediction is not music then follow the MusicType branch to predict type of Music</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
 def __init__(self):
     super(Encoder, self).__init__()
     self.enc = nn.Encoderlayers()   #Some N number of encoder layers
 def forward(self, x):
     return self.enc(x)       
</code></pre>
<pre><code class="lang-auto">class isMusicNet(nn.Module):
 def __init__(self):
     super(isMusicNet, self).__init__()
     self.fc = nn.Linear(64, 2)
 def forward(self, x):
     return self.fc(x)
</code></pre>
<pre><code class="lang-auto">class MusicTypeNet(nn.Module):
 def __init__(self):
     super(MusicTypeNet, self).__init__()
     self.fc = nn.Linear(64, 4)
 def forward(self, x):
     return self.fc(x)
</code></pre>
<pre><code class="lang-auto">class OverallNet(nn.Module):
 def __init__(self):
     super(MusicTypeNet, self).__init__()
     self.enc = Encoder()
     self.musicnet = isMusicNet()
     self.musictypenet = MusicTypeNet()
 def forward(self, x):
     x = self.enc(x)
     x_m = self.musicnet(x)
     x_mtype = self.musictypenet(x)
     return x_m, x_mtype
</code></pre>
<p>Now Consider, I have 5 classes in total<br>
C1, C2, C3, C4, C5<br>
Suppose C5 is non music<br>
Where 4 classes are of 4 different types of music and other one is not music<br>
What I want is to have my network , two branches, one branch differentiates between music and non music [i. e,  it will be having two classes] and other predicts or differentiates between music types [in this case should have 4 classes]</p>
<p>I am getting confused in the training loop.<br>
Since in <code>isMusicNet</code>, I should have binary cross entropy loss<br>
and in <code>MusicTypeNet</code>, I should be having categorical cross entropy loss, to differentiate between 4 classes<br>
but when the pytorch loader passes the batches, it contains both music and non music samples. How to mask the non music one in <code>MusicTypeNet</code> and combine  music types as one class in <code>isMusicNet</code></p>
<p>In addition, if the label are</p>
<blockquote>
<p>batchlabels = tensor([4, 4, 2, 4, 4, 0, 3, 4, 0, 1, 4, 0, 3, 2, 4, 4, 1, 4, 4, 2, 4, 0, 4, 0,<br>
0, 0, 0, 4, 0, 4, 4, 4, 4, 4, 4, 4, 0, 0, 4, 4, 1, 4, 4, 1, 4, 0, 4, 1,<br>
4, 4, 1, 4, 4, 3, 3, 3, 3, 4, 1, 0, 4, 0, 0, 4])</p>
</blockquote>
<p>and I want</p>
<blockquote>
<p>batchlabels[batchlabels!=5]<br>
tensor([2, 0, 3, 0, 1, 0, 3, 2, 1, 2, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 3,<br>
3, 3, 3, 1, 0, 0, 0])  as one class suppose class A.</p>
</blockquote>
<p>and</p>
<blockquote>
<p>batchlabels[batchlabels==5]<br>
tensor([4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4,<br>
4, 4, 4, 4, 4, 4, 4, 4, 4])<br>
as class B</p>
</blockquote>
<p>Since my original labels are 5 in dataloader, how can I map these newly created label in the loss function now</p>
<p>Any help or suggestion would be appreciated ?<br>
Thank you</p>
How to retrain the trained model with new images(not a pretrained model)?

Revathi_R: <p>I am creating the pytorch model.pt with few images initially. Again I am trying to train with some more images. because of the previously trained model not giving the exact result as I expect. but the model starts training with previously added images and new images because of this it takes more time to train the model.<br>
here I want to know how to retrain the model with new images alone. Also should avoid old images getting trained again. How to combine already trained with newly trained images as single model(purpose to avoid training time of already trained images).<br>
will this possible?</p>
PyTorch causes zombie processes on multi-GPU system

ATAboukhadra: <p>Hi,</p>
<p>I’m running my PyTorch code on a GPU on a remote server. The server has 8 GPUs and they are shared among multiple users. So I run my code on 1 GPU out of them using DataParallel class. Whenever I try to terminate the process it turns into a zombie process as you can see in the nvidia-smi output here:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/41efdf0b4d6f7acab7ffd509c649b7b84214a91f.png" data-download-href="https://discuss.pytorch.org/uploads/default/41efdf0b4d6f7acab7ffd509c649b7b84214a91f" title="Screen Shot 2021-12-23 at 1.38.18 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41efdf0b4d6f7acab7ffd509c649b7b84214a91f_2_383x500.png" alt="Screen Shot 2021-12-23 at 1.38.18 PM" data-base62-sha1="9piXmetMgkqYkXlRzwSgq1d8pjN" width="383" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41efdf0b4d6f7acab7ffd509c649b7b84214a91f_2_383x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41efdf0b4d6f7acab7ffd509c649b7b84214a91f_2_574x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/41efdf0b4d6f7acab7ffd509c649b7b84214a91f_2_766x1000.png 2x" data-dominant-color="292929"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2021-12-23 at 1.38.18 PM</span><span class="informations">1116×1456 251 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>All processes with a dash as the process name are killed processes that are still occupying the GPU and can’t be removed unless the system is rebooted which affects all users of the system.</p>
<p>I can’t produce this problem on my machine that has only 1 GPU and the code runs normally with no problems and terminates properly. This problem occurred on the multi-GPU system just last week and I’m not sure whether this was caused by a change in the PyTorch library or is a problem from my side.</p>
<p>If you can give me a fix or check what the problem might be it would be great.</p>
<p>Kind regards,<br>
Ahmed</p>

VFAndy: <p>Running into the same issue here, Pytorch creates Deadlocking processes (I debugged into it, let me know if you want details where the dealock happens – seems to be a threading issues imo) which become zombies and use up a lot of GPU RAM while at the same time blocking any further or current GPU jobs. I will try adding the details (Pytorch version, cuda version etc.) later.</p>
Applying collate_fn for multiple interleaved Datasets

Groot: <p>I have two datasets say,<br>
MyFirstDataset =MyFirstDatasetClass(data1)<br>
MySecondDataset = MySecondDatasetClass(data2)<br>
I have written a sampler which interleaves the sampling from two datasets which take input the concatenated dataset<br>
concat_dataset = ConcatDataset([MyFirstDataset,MySecondDataset])<br>
and give me the interleaved schedule. But the problem is and I am loading a dataset using a single data loader</p>
<p>dataloader  = DataLoader(dataset = concat_dataset, sampler = BatchSchedulerSampler(dataset = concat_dataset,batch_size = batch_size),								batch_schedule = args.batch_schedule),batch_size = batch_size)<br>
I want to apply separate collate_fn for the two datasets but the dataloader only allows one collate_fn(correct me if I am wrong), please let me know how apply separate collate_fn in the dataloader if the dataset consists of 2 or more datasets.</p>

FarisHijazi: <p>return some type of identifier from the dataset, indicating if it’s dataset <span class="hashtag">#1</span> or <span class="hashtag">#2</span>.<br>
Then in the collate function just group them and do your thing</p>
How to pass video frames to cnnlstm for classification/regression

talhaanwarch: <p>I have a video with shape of <code>torch.Size([1,129, 3, 224, 224])</code> where 1 is batch size, 129 is number of frames, 3 is channel and 224 is the width height dimension.<br>
I want to do classification/regression task. Here can use a 3DCNN or 2DCNN with LSTM. (Let me know if there is any other option).<br>
First i am starting with 2D CNN with LSTM<br>
First is to  remove the batch index using <code>i=i[0]</code>. This make the frame index as batch index.<br>
After that i split the frames into batches <code>i=i[len(i)%32::].reshape(len(i)//32,-1,3,224,224)</code><br>
Goal here is to take first 32 frames as first batch, next 32 frames as next batch, and so on. Let  me know if reshaping is correct. The shape is <code>[4, 32, 3, 224, 224]</code><br>
Then i pass each batch to the model (Resnet).</p>
<pre><code class="lang-auto">x1=[]#features
        for x in i:
            x1.append(self.model(x))
        x=torch.stack(x1)
</code></pre>
<p>Now i got a shape <code>[4, 32, 1000]</code>. I want to pass this to lstm layer, but i dont know.<br>
So, i create a layer</p>
<pre><code class="lang-auto">lstm = nn.LSTM(input_size=1000, hidden_size=512, batch_first=True)
o,(h,c)=lstm(x)
o.shape,h.shape,c.shape
(torch.Size([4, 32, 512]), torch.Size([1, 4, 512]), torch.Size([1, 4, 512]))
</code></pre>
<p>Now i can consider the first index as batch and keeping this in mind, i can replicated my single ground label to an array of four and do classification/regression.</p>
<p>But i am not sure above logic or syntax is correct</p>
How to define Dataset multiple image inputs and Masks for an ensemble model for semantic segmentation task?

TANMAY_KUMAR: <p>My model is as follows:</p>
<pre><code>class Ensemble(nn.Module):
def __init__(self, model0, model1, model2, n_classes = 0, activation='relu'):
    super(Ensemble, self).__init__()
    self.model0 = model0
    self.model1 = model1
    self.model2 = model2
    self.n_classes = n_classes
    self.activation = Activation(activation)
    
    self.up1 = Up(3072, 768)
    self.up2 = Up(768, 192)
    self.up3 = Up(192, 48)
    self.up4 = Up(48, 12)
    self.out_final = nn.Conv2d(12, n_classes, 1)
    
    
def forward(self, x1, x2, x3):
    x1 = self.model0(x1)
    x2 = self.model1(x2)
    x3 = self.model2(x3)
    x = torch.cat((x1, x2), dim=1)
    x = torch.cat((x, x3), dim=1)  #(End of the encoder)
    
    out = self.up1(x)
    out = self.up2(out)
    out = self.up3(out)
    out = self.up4(out)
    out = self.out_final(out)
    out = self.activation(out)
    out = F.softmax(out, dim=1)
    return out
</code></pre>
<p>How to define Dataset multiple image inputs and Masks for the above ensemble model for semantic segmentation task?</p>
Simple variational net with torch.distributions

ivallesp: <p>Hi there,</p>
<p>I am trying to build a <strong>toy variational network</strong> that, sampling from a learned Gaussian posterior distribution, approximate another unknown distribution through a series of fully connected layers. In the code below, the target distribution is defined as a N(4,3).</p>
<p>I am trying to do that with variational inference. However, my model does not learn the target distribution at all, in fact it does learn to ignore posterior in the input, predicting always a constant value for different samples of the posterior. I don’t want to use Maximum Likelihood because the target distribution is pretended to be unknown.</p>
<p>Can anyone help me understand (1) why is that happening and (2) how to do this correctly? Reproducible code below.</p>
<pre><code class="lang-auto">import numpy as np
import matplotlib.pyplot as plt
import torch

# Define the parameters of the target Gaussian distribution we want to approximate
# This distribution is meant to be unknown (e.g. an image)
MU = 4
SIGMA = 3

# Define a neural network that learns the parameters of a posterior (mu and sigma), samples
# from that distribution, and produces a scalar output after a series of projections.

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        # Prior for the KLD
        self.prior = torch.distributions.Normal(0, 1);
        # Gaussian posterior distribution parameters to be learned by the model
        self.mu = torch.nn.Parameter(torch.normal(0, 0.1, size=(1,)), requires_grad=True)
        self.var = torch.nn.Parameter(torch.ones(1), requires_grad=True)
        self.posterior = torch.distributions.Normal(self.mu, self.var)
        # Multi-Layer Perceptron 
        self.projection_1 = torch.nn.Linear(in_features=1, out_features=10)
        self.projection_2 = torch.nn.Linear(in_features=10, out_features=10)
        self.projection_out = torch.nn.Linear(in_features=10, out_features=1)

    def forward(self):
        # Sample from the posterior
        z = self.posterior.sample()
        # MLP Projection
        h = torch.nn.ReLU()(self.projection_1(z))
        h = torch.nn.ReLU()(self.projection_2(h))
        # Output projection
        o = self.projection_out(h)
        return o

net = Net()
criterion = torch.nn.MSELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.01)

# Do 50,000 steps of gradient descent

for i in range(50000):
      net.zero_grad()  # Reset the gradient
      # Forward
      y_hat = net()  
      # Take a sample from the target distribution
      y = torch.normal(MU, SIGMA, size=(1,)) 
      # Calculate the KLD
      kld = torch.distributions.kl_divergence(net.prior, net.posterior)
      # Calculate Loss
      loss = criterion(y_hat, y) + kld/100
      # Backward
      loss.backward() 
      # Training step
      optimizer.step()

# Now let's inspect the learned distribution
outputs = [net().cpu().detach().numpy().item() for _ in range(1000)]
plt.hist(outputs)

</code></pre>
Does torch.Tensor.view affect the backward step?

dungxibo123: <p>Hi,</p>
<p>I know my title is kind of weird. Here is the problem.</p>
<p>In my model, when implement the forward step. I use <code>torch.Tensor.view()</code> method.<br>
Reasons: the <a href="https://github.com/google-research/torchsde/" rel="noopener nofollow ugc"><code>torchsde</code></a> does not allow the k-D input tensors (k &gt; 2).</p>
<pre><code class="lang-python">    def forward(self,x):
        out = self.fe(x)
        bs = x.shape[0]
#        print(f"Shape after Feature Extraction Layer: {out.shape}")
        out = out.view(bs,-1)
#        print(f"After the feature extraction step, shape is: {out.shape}")
#        print(f"Device of out {out.device}")
#        print(f"Shape before the SDE Intergral: {out.shape}")
        out = sdeint(self.rm,out,self.intergrated_time, options=self.option,method="euler", atol=5e-2,rtol=5e-2, dt=0.1, dt_min=0.05)[-1]
        out = out.view(bs,self.input_conv_channel, self.input_conv_size, self.input_conv_size)
        out = self.fcc(out)
        return out
</code></pre>
<p>I have view the tensor as flattened shape, after that i apply the <code>sdeint</code> to the data.<br>
But for some reason, the accuracy is really bad on any size dataset (cifar10 and svhn).</p>
<p>I also tried a same method (<code>sdeint</code>) for bunch of linear layers, which I don’t need to use <code>view</code> method to forward. And the accuracy is really good.</p>
<p>I have read and see the document say that<br>
" PyTorch allows a tensor to be a <code>View</code> of an existing tensor. View tensor shares the same underlying data with its base tensor. Supporting <code>View</code> avoids explicit data copy, thus allows us to do fast and memory efficient reshaping, slicing and element-wise operations.".<br>
But I am in doubt that the <code>view</code> method affected my training process.  Doesn’t it ?.<br>
Hope to received the answer.</p>
<p>Many thanks<br>
Best regards,<br>
Tien Dung</p>
Multipy two tensors

Umair_Javaid1: <p>I want to multiply two tensors x and y, such that z is the output. How to do this?</p>
<p>X = torch.tensor([[[1,2,3,4],[5,6,7,8],[9,9,9,9]],[[10,20,30,40],[50,60,70,80],[90,90,90,90]]])</p>
<p>y = torch.tensor([[10,100,1000],[2,4,6]])</p>
<p>X = torch.tensor([[[10,20,30,40],[500,600,700,800],[9000,9000,9000,9000]],[[20,40,60,80],[200,240,280,320],[540,540,540,540]]])</p>

AlphaBetaGamma96: <p>Hi,</p>
<p>the following will get what you want,</p>
<pre><code class="lang-auto">Z = X*y.unsqueeze(2)
</code></pre>
<p>Which are equal,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; X*y.unsqueeze(2)
tensor([[[  10,   20,   30,   40],
         [ 500,  600,  700,  800],
         [9000, 9000, 9000, 9000]],

        [[  20,   40,   60,   80],
         [ 200,  240,  280,  320],
         [ 540,  540,  540,  540]]])
&gt;&gt;&gt; Z
tensor([[[  10,   20,   30,   40],
         [ 500,  600,  700,  800],
         [9000, 9000, 9000, 9000]],

        [[  20,   40,   60,   80],
         [ 200,  240,  280,  320],
         [ 540,  540,  540,  540]]])

</code></pre>
No module named 'torch' linux fedora 34

herlimenezes: <p>I am doing some test using RNN for forecasting, and I am using Python version 3.9.9. A virtual environment was created using pip3 and is named as RedesNeuraisEnv. When I try to use it in Jupyter notebook I get this message: No module named ‘torch’<br>
List of packages installed in that environment is at <a href="https://paste.centos.org/view/ce7bfc9a" class="inline-onebox" rel="noopener nofollow ugc">UNTITLED - Pastebin Service</a></p>
<p>How can I solve this?</p>
Skipping argmax during backpropagation

abedshantti: <p>I am creating a GAN that can also utilize an auxiliary classifier from an external library for computing an additional loss that can be backpropagated to the GAN. The auxiliary classifier is a LightGBM model, which was chosen due to its computational efficiency. The input to LightGBM is typically a panda DataFrame, where each feature is represented in a single dimension.</p>
<p>Since the output of my generator is a one-hot encoded model, I have to first apply <code>torch.argmax(cat_output)</code> to make the output compatible for the LightGBM model. Problems arise when backpropagating the loss, as the argmax detaches the gradients from the output. Is there a way to skip the argmax when backpropagating the loss, as the argmax merely exists for making the output compatible for use in the alternative model, whereas the learning is required for the layers before it.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a5c9ff9e3f50d033a2c9dacb41b0fd405f7ed8b.png" data-download-href="https://discuss.pytorch.org/uploads/default/3a5c9ff9e3f50d033a2c9dacb41b0fd405f7ed8b" title="pytt"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/a/3a5c9ff9e3f50d033a2c9dacb41b0fd405f7ed8b.png" alt="pytt" data-base62-sha1="8ki8KTUPGGGvRYoNLsv7TdXomzF" width="195" height="500" data-dominant-color="F4F1EC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytt</span><span class="informations">254×651 4.45 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Equivalent functions but different gradients

HLSS: <p>Detailed description in  <a href="https://github.com/pytorch/pytorch/issues/70578#" rel="noopener nofollow ugc">issue 70578</a></p>
<hr>
<p>However, in the attempt of the following code, it seems that the correct gradient is obtained.</p>
<p>I wonder if anyone can help me solve this problem or give me some debugging suggestions</p>
<p><a href="https://gist.github.com/HLSS-Hen/6d77586c4ed455f7fb3906f438e630ae" rel="noopener nofollow ugc">code</a></p>
Gradient flow in layers

cbd: <p>I have sequence of series connection of “conv(3x3)-norm-leakyrelu”-“dialated_conv(3x3)-norm” with short skip connection(like Resnet) followed by last layer of “conv(1x1)-tanh”. I train model for only 2 images which gives below histogram of weight and gradients. One loss out of 2 is stuck after some epochs. All the layers before “layer_10” has histogram like “layer_10” which i think these layers are not learning. I use the xavier initialisation with initial gain 0.15.  Also note the values of weight and gradients on x-axis. I think i am missing something after looking in histogram.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51ca00b167da762b4dbcc6b1521cdbd1c2173a96.png" data-download-href="https://discuss.pytorch.org/uploads/default/51ca00b167da762b4dbcc6b1521cdbd1c2173a96" title="hist"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51ca00b167da762b4dbcc6b1521cdbd1c2173a96_2_690x387.png" alt="hist" data-base62-sha1="bFxsLgLG18KWbfcbJmpFFXyPZEa" width="690" height="387" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/1/51ca00b167da762b4dbcc6b1521cdbd1c2173a96_2_690x387.png, https://discuss.pytorch.org/uploads/default/original/3X/5/1/51ca00b167da762b4dbcc6b1521cdbd1c2173a96.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/1/51ca00b167da762b4dbcc6b1521cdbd1c2173a96.png 2x" data-dominant-color="FAEDE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">hist</span><span class="informations">819×460 56.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Convert a Binary GumbelSoftmax output to BCE

abedshantti: <p>I am trying to calculate the loss between a feature GAN output and the labels produced by an auxiliary classifier (AC). For the GAN, I am using tanh activations for numerical features and gumbel softmax for categorical features. The AC can calculate the MSE for numerical features, cross entropy for categorical features and binary cross entropy (BCE) for binary features.</p>
<p>Since my GAN uses gumbel softmax for all non-numerical features, I am unable to compute the loss between the GAN gumbel softmax output (2 units) and the AC BCE output (1 unit, similar to using a single output node with sigmoid activation). Is there a way that I can convert my gumbel softmax output to a single probability vector that can be used by the AC.</p>
<p>The AC is built using an external library (LightGBM), so I am trying to modify pytorch’s output to fit there. Also, using the CE for binary features is not recommended as stated in the LightGBM documentation.</p>
Seq2Seq: Get first element of every document in a batch

Lady_Hangaku: <p>For an encoder decoder task, I want to get every first element in the whole batch to feed it into the decoder (first element of every document is always the  tag).<br>
Let’s say I have this output tensor:</p>
<pre><code class="lang-auto">tensor([[   1, 1870, 1871,  ...,    0,    0,    0],
        [   1, 1159, 1160,  ...,    0,    0,    0],
        [   1, 1159, 1500,  ..., 1651, 1170, 1167],
        [   1, 1172, 1173,  ...,    0,    0,    0]])
</code></pre>
<p>How can I get:</p>
<pre><code class="lang-auto">tensor([[   1  ],
        [   1  ],
        [   1  ],
        [   1  ]])
</code></pre>
<p>I am following <a href="https://towardsdatascience.com/a-comprehensive-guide-to-neural-machine-translation-using-seq2sequence-modelling-using-pytorch-41c9b84ba350#ebac" rel="noopener nofollow ugc">this tutorial</a> and the part which doesn’t work for me is under “9. Seq2Seq (Encoder + Decoder) Code Implementation” in line 22.<br>
Has anyone an idea? Thank you!</p>

ptrblck: <p>I’m not sure I understand the use case completely, but wouldn’t indexing the first column work?</p>

Lady_Hangaku: <p>Yes I think so, I just don’t know how to do that.<br>
But I think I got the right solution (<code>target_tensor</code> is the given tensor above):<br>
<code>target_tensor[torch.arange(target_tensor.size(0)), 0]</code></p>

ptrblck: <p>Your approach would work but alternatively you could also use <code>target_tensor[:, 0]</code>.</p>

Lady_Hangaku: <p>Ah thank you, that is way easier!</p>
Pytorch live to media pipe calculators

Prince_Patel: <p>I am trying to build a pytorch inference calculator. A sample calculator for tensorflow-lite is present at <a href="https://github.com/google/mediapipe/tree/master/mediapipe/calculators/tflite" class="inline-onebox" rel="noopener nofollow ugc">mediapipe/mediapipe/calculators/tflite at master · google/mediapipe · GitHub</a></p>
<p>What all dependencies are needed for this?</p>
A proper latent space dimension for a conditional VAE model

Minsu_Kim: <p>Hello, I have a question about a conditional VAE model (CVAE model).<br>
My CVAE model generates a 2D-sample (x,y), (=data, X) in a grid map.<br>
The latent space (=z) dimension is 2, and the dimension of the conditional features (grid map features) is 8. The conditional features are an embedded grid map by CNN.<br>
It runs well as I expected at test environments</p>
<p>Here is my question.</p>
<p>Is it OK with setting the latent space dimension (=z) as the same as the data dimension (=X)?<br>
As I know, the objective of the VAE model is ‘to generate something with sampled latent vector’, not to reduce the dimension of data (=X), such as AE.<br>
Or is there any other theoretic assumption that I need to set latent space dimension is lower than data dimension?</p>
<p>Thanks,</p>
DLL load failed on window10 with latest version

Nathan_Drake: <pre><code class="lang-auto">ImportError                               Traceback (most recent call last)
&lt;ipython-input-7-ce8d15220a2c&gt; in &lt;module&gt;
----&gt; 1 import torch
      2 
      3 
      4 print(torch.__version__)

c:\users\nathan\pytorch\torch-env\lib\site-packages\torch\__init__.py in &lt;module&gt;
     79 del _dl_flags
     80 
---&gt; 81 from torch._C import *
     82 
     83 __all__ += [name for name in dir(_C)

ImportError: DLL load failed: 找不到指定的模块。
</code></pre>
<p>I have created a virtualenv using python 3.7 (prue python) which contains torch1.2<br>
I got this problem after I update torch1.3 (just pip install and did not uninstall torch1.2)</p>
<p>After I delete all python related file(python file, pip cache) and reinstalled I still got this error. I think maybe it is a version related problem?</p>

Nathan_Drake: <p>My solution is to use conda.</p>

evanmirk: <p>To make it short, it means that you lacked some “dependencies” for the libraries you wanted to use.  This is a common problem when installing python packages, mainly in windows. Before trying to use any kind of library, first it is suggested to look up whether it needs another library in python “family”.</p>
<p>The solution is to provide the python interpreter with the path-to-your-module/library. The simplest solution is to append that <a href="http://net-informations.com/python/intro/path.htm" rel="noopener nofollow ugc">python path</a> to your sys.path list. In your notebook, first try:</p>
<pre><code class="lang-auto">import sys
sys.path.append('my/path/to/module/folder')
</code></pre>
<p>This isn’t a permanent change in sys.path, because when you log out, your environment is reset, so any variables you may have set are lost.</p>
<p>The better (and more permanent) way to solve this is to set your PYTHONPATH, which provides the interpreter with additional directories look in for python packages/modules.</p>
<p><code>from BASH type: export PYTHONPATH=/path/to/new/folder:/another/path/...../       #each path must be separated by a colon</code></p>
Installing PyTorch on macOS Big Sur

rohgaj: <p>Hi,<br>
I have a macOS Big Sur laptop that is Version 11.6.1, and I need help trying to figure out how to install PyTorch on my computer.<br>
Warm Regards,<br>
Rohan Gajraj</p>

JamesWright: <aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/get-started/locally/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://pytorch.org/assets/images/pytorch-logo.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://pytorch.org/get-started/locally/" target="_blank" rel="noopener nofollow ugc">PyTorch</a></h3>

  <p>An open source machine learning framework that accelerates the path from research prototyping to production deployment.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


rohgaj: <p>I don’t have Python installed on my laptop. What version of Python would I need for PyTorch?</p>

JamesWright: <p><a class="mention" href="/u/rohgaj">@rohgaj</a> You can find that information on the page I’ve linked in previous reply.</p>

rohgaj: <p>Thanks for your help.</p>

rohgaj: <p>Hi,<br>
I have installed Python via the Python website, and pip was installed along with it. At the moment, I’m a little stuck trying to figure out how to install PyTorch using pip?</p>
How to save gradients for next iteration step

Eduin_Hernandez: <p>Hello, I’m trying to write a custom autograd function where I sparsify the gradients but keep the non-sparsified portion for the next iteration step as an error compensation just like in the paper “<a href="https://arxiv.org/abs/1809.07599" rel="noopener nofollow ugc">Sparsified SGD with Memory</a>”.</p>
<p>I know I can use ctx to store information from the forward propagation to backpropagation, but what I need is to store it from the current to the next backpropagation instance. Is there a way to do so?</p>
<p>I previously tried to use hooks in order to rewrite the backprogation behavior and store inside the class these gradients, but it is unstable for Conv2d when using CUDA.</p>
DistributedSampler and Subset() data duplication with DDP

pysam: <p>I have a single file that contains N samples of data that I want to split into train and val subsets while using DDP. However, I am not entirely sure I am going about this correctly because I am seeing replicated training samples on multiple processes. To reproduce the issue I created a simple dataset class. Here’s my current way of doing this  -</p>
<p>class TrivialDataset(Dataset):</p>
<pre><code>def __init__(self,N):
    self.length = N

def __len__(self):
    return self.length

def __getitem__(self, index):
    return index
</code></pre>
<p>The above class is for illustration purposes only - this lets me print the indices retrieved on each DDP process.</p>
<p><code>dataset = TrivialDataset(20)</code><br>
<code>train_subset, val_subset = torch.utils.data.random_split(dataset, [16, 4])</code><br>
<code>train_dataset = Subset(dataset, train_subset)</code><br>
<code>val_dataset = Subset(dataset, val_subset) </code></p>
<p>I can confirm that the train and val datasets above have independent indices. Next, I create a different sampler for the train and val subsets :</p>
<p><code>seed = 10</code><br>
<code>train_sampler = DistributedSampler(train_dataset, num_replicas=world_size, rank=global_rank,                                 shuffle=False, seed=seed, drop_last=True)</code></p>
<p><code>val_sampler = DistributedSampler(val_dataset, num_replicas=world_size, rank=global_rank,                                  shuffle=False, seed=seed, drop_last=True)</code></p>
<p>Finally, I create the data loaders for each subset<br>
<code>train_loader = DataLoader(train_dataset, batch_size=batch_size, sampler=train_sampler) </code><br>
<code>val_loader   = DataLoader(val_dataset, batch_size=batch_size, sampler=val_sampler) </code></p>
<p>Now if I iterate over the train_loader on 2 pytorch DDP processes (and print the indices retrieved by each <code>train_loader</code>), I see duplicates on the two processes. I would expect that each process would have independent indices. The expectation is that the train_subset indices and val_subset indices would be independently split across 2 processes, but that’s not what is happening. The ultimate goal is to scale up across N&gt;&gt;2 processes. I’m not sure where I am going wrong. If I don’t split the indices into train and val subsets, the code works as expected. I’ve searched this forum but haven’t come across a similar issue.<br>
Thanks!</p>

mrshenli: <aside class="quote no-group" data-username="pysam" data-post="1" data-topic="140317">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/ac91a4/48.png" class="avatar"> pysam:</div>
<blockquote>
<p>DistributedSampler</p>
</blockquote>
</aside>
<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> for DistributedSampler questions</p>