akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Calculating upto more decimal places by "linalg.eigh"

pankajsharma6810: <p>A = torch.randn(2, 2, dtype=torch.complex128)<br>
A = A + A.T.conj()  # creates a Hermitian matrix<br>
A<br>
L, Q = torch.linalg.eigh(A)<br>
L</p>
<p>The above code gives the eigenvalues and elements of eigenvectors up to 4 decimal places. How to get more decimal places? Please help.</p>

KFrank: <p>Hi Panka!</p>
<aside class="quote no-group" data-username="pankajsharma6810" data-post="1" data-topic="173105" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pankajsharma6810/48/56839_2.png" class="avatar"> pankajsharma6810:</div>
<blockquote>
<p>The above code gives the eigenvalues and elements of eigenvectors up to 4 decimal places.</p>
</blockquote>
</aside>
<p>By default, to save screen space, pytorch only displays tensors with four digits.<br>
Try <code>.item()</code> or <code>torch.set_printoptions()</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.13.1'
&gt;&gt;&gt; _= torch.manual_seed (2023)
&gt;&gt;&gt; A = torch.randn(2, 2, dtype=torch.complex128)
&gt;&gt;&gt; A = A + A.T.conj() # creates a Hermitian matrix
&gt;&gt;&gt; A
tensor([[-1.7077+0.0000j, -0.7976+0.3775j],
        [-0.7976-0.3775j,  0.1020+0.0000j]], dtype=torch.complex128)
&gt;&gt;&gt; L, Q = torch.linalg.eigh(A)
&gt;&gt;&gt; L
tensor([-2.0667,  0.4610], dtype=torch.float64)
&gt;&gt;&gt; L[0].item()
-2.0667332207652334
&gt;&gt;&gt; torch.set_printoptions (precision = 15)
&gt;&gt;&gt; L
tensor([-2.066733220765233,  0.461039667016129], dtype=torch.float64)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Diagonalizing a 40k by 40k matrix

pankajsharma6810: <p>torch.linalg.eigh(matrix) , this function is unable to find the eigenvectors of a 40K by 40 K matrix? Is there any solution to that. The matrix is sparse, but I need full eigenvectors. The numpy.linalg.eigh() is also not working. Only solution is scipy.linalg.eigsh() but this is costly in time, about 10 hours for one diagonalization process.<br>
Please suggest something fast and reliable.</p>
Can't get the whole gradients of conv1

big_old: <p>Hi, I am trying to get the gradients when I train Resnet50 model provided by DeepLearningExamples. And the problem is successfully solved by calling this function “print(self.model.module.conv1.weight.grad)”. But I found there are too many ‘…’ in gradients. Although I set “np.set_printoptions(threshold=sys.maxsize)”, this problem is still not solved. In addition to this, I found self.model.module.conv1.weight.grad is a NoneType object and I can’t use some torch functions.<br>
So how can I get the whole gradients？<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd4a0e1b377d83ce09c25d2a742dd33e30b8b0d5.png" data-download-href="https://discuss.pytorch.org/uploads/default/bd4a0e1b377d83ce09c25d2a742dd33e30b8b0d5" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bd4a0e1b377d83ce09c25d2a742dd33e30b8b0d5_2_690x276.png" alt="image" data-base62-sha1="r0wU3TURoDwBsrxmy1vwJr6hyrb" width="690" height="276" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/d/bd4a0e1b377d83ce09c25d2a742dd33e30b8b0d5_2_690x276.png, https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd4a0e1b377d83ce09c25d2a742dd33e30b8b0d5.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd4a0e1b377d83ce09c25d2a742dd33e30b8b0d5.png 2x" data-dominant-color="E7E4E4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">952×382 89.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Using <code>np.set_printoptions</code> won’t change the behavior of PyTorch and you should use <code>torch.set_printoptions(threshold=...)</code> instead.</p>
<p>The <code>.grad</code> attributes are <code>None</code> if they were not computed and will be filled during the <code>backward</code> call assuming the corresponding parameter was used to calculate the model output.</p>

big_old: <p>Thank you very much! I figure it out by setting torch.set_printoptions(threshold=sys.maxsize)</p>
Multiprocess dataloader using TensorDataset

zhangzp9970: <p>Hello! I use the following code to load a tensordataset from a pkl file. However, when I set the data workers in DataLoader to 2, I will encounter a GPU error<br>
“[W CUDAGuardImpl.h:62] Warning: CUDA warning: invalid device ordinal (function uncheckedSetDevice)<br>
[W CUDAGuardImpl.h:46] Warning: CUDA warning: driver shutting down (function uncheckedGetDevice)”<br>
How to solve this problem?</p>
<pre><code class="lang-auto">   train_image=torch. Load(train_image_dir)
   train_label=torch.load(train_label_dir)
   test_image=torch.load(test_image_dir)
   test_label = torch.load(test_label_dir)

   train_ds = TensorDataset(train_image, train_label)
   test_ds = TensorDataset(test_image, test_label)

   train_ds_len = len(train_ds)
   test_ds_len = len(test_ds)

   print(train_ds_len)
   print(test_ds_len)

   # for train
   train_dl = DataLoader(dataset=train_ds, batch_size=batch_size,
                         shuffle=True, num_workers=data_workers, drop_last=True)
   # for evaluate
   test_dl = DataLoader(dataset=test_ds, batch_size=batch_size,
                        shuffle=False, num_workers=data_workers, drop_last=False)
</code></pre>

ptrblck: <p>Based on the described behavior I would guess your <code>Dataset</code> is trying to push tensors directly to a GPU and is using an invalid device index. Could you check the stacktrace and narrow down where this operation is performed and post the corresponding code here, please?</p>
Question about EfficientDET layer input / output size mismatch

Lu_Bin_Liu: <p>I have a question about EfficientDET model <a href="https://raw.githubusercontent.com/mrdbourke/pytorch-deep-learning/main/images/06-torchinfo-summary-unfrozen-layers.png" rel="noopener nofollow ugc">here</a>. Why does the AdaptiveAVG2D layer have am output size of (32, 1280, 1, 1), yet the input size of the succeeding classifier layer is (32, 1280)? I thought that the two had to match, or is there some kind of flattening that was done internally?</p>
<p>The code that generated the figure is below:</p>
<blockquote>
<p>weights = torchvision.models.EfficientNet_B0_Weights.DEFAULT weights<br>
model = torchvision.models.efficientnet_b0(weights=weights).to(device)<br>
summary(model=model,<br>
input_size=(32, 3, 224, 224),<br>
col_names=[“input_size”, “output_size”, “num_params”, “trainable”],<br>
col_width=20,<br>
row_settings=[“var_names”]<br>
)</p>
</blockquote>
For loop traversing the dataset causes memory corruption

goodboy-hub: <p>用 for loop 去遍历dataset 会把整个datasat加载到内存导致崩溃吗？我在一次机器学习任务中出现了内存崩溃，我怀疑是for在捣鬼</p>

ptrblck: <p>From Google translate:</p>
<blockquote>
<p>Will using a for loop to traverse the dataset load the entire datasat into memory and cause a crash? I have a memory crash in a machine learning task, I suspect that for is playing tricks</p>
</blockquote>

goodboy-hub: <p>Thanks for reply,my problem have been solved</p>
Autograd Static Error

Raphael_Olipitz: <p>Hey Guys,</p>
<p>hope you are doing well. I want to use a very cool loss function from a new paper and its using autograd with a staticmethod like:</p>
<p>class RankSort(torch.autograd.Function):<br>
<span class="mention">@staticmethod</span><br>
def forward(ctx, logits, targets, delta_RS=0.50, eps=1e-10):</p>
<pre><code>    classification_grads=torch.zeros(logits.shape).cuda()
    
    #Filter fg logits
    fg_labels = (targets &gt; 0.)
    fg_logits = logits[fg_labels]
    fg_targets = targets[fg_labels]
    fg_num = len(fg_logits)
</code></pre>
<p>ect ect.</p>
<p>But if I use the function in my traningsloop like:</p>
<pre><code> rank_sort = RankSort(k=3)
</code></pre>
<p>…</p>
<pre><code>   ranking_loss, sorting_loss = rank_sort(logits, y,delta_RS =0.5)
    classification_loss = criterion(logits, y)
    loss = classification_loss + ranking_loss + sorting_loss
</code></pre>
<p>I got the following error:</p>
<p>RuntimeError: Legacy autograd function with non-static forward method is deprecated. Please use new-style autograd function with static forward method. (Example: <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 1.13 documentation</a></p>
<p>I do not know why because as I understood it I using a static method.</p>
<p>Thank for the Help!</p>

ptrblck: <p>It seems the <code>.apply</code> operation is missing in your code as indicated by the linked tutorial.</p>
Portability of forward() with custom input

perf: <p>Hi,</p>
<p>Assume that I have defined a C++ libtorch model that performs forward() with a custom struct as input. For example:</p>
<pre><code class="lang-auto">void forward(custom_d input){
return input.a + input.b
}

</code></pre>
<p>I want to save the model and use it in Python via torch.jit.load, is it possible to inference the pre-trained model since I don’t have this data structure (input) in my python program?</p>
Using DDP with WebDataset

adhakal224: <p>This is my first time using WebDataset and I have multiple shards (about 60) with a large number of images. It was working as I would expect in the normal <code>Dataset</code> class when I was using a single GPU. However once I set the <code>devices</code> to 2 I received the error <code>ValueError: you need to add an explicit nodesplitter to your input pipeline for multi-node training Webdataset</code>.<br>
I saw two approach to allow using multiple gpus with <code>WebDataset</code>.</p>
<ol>
<li>Using <code>.with_epochs</code>
</li>
</ol>
<p>According to <a href="https://github.com/webdataset/webdataset" rel="noopener nofollow ugc">WebDataset Github</a> I could simply use the <code>with_epochs</code> function in my dataset as follows:</p>
<pre><code class="lang-auto">dataset = wds.WebDataset(url, resampled=True).shuffle(1000).decode("rgb").to_tuple("png", "json").map(preprocess).with_epoch(10000) 
dataloader = wds.WebLoader(dataset, batch_size=batch_size)
</code></pre>
<ol start="2">
<li>Using <code>ddp_equalize</code><br>
According to <a href="https://webdataset.github.io/webdataset/multinode/" rel="noopener nofollow ugc">WebDataset MultiNode</a>
</li>
</ol>
<pre><code class="lang-auto">dataset_size, batch_size = 1282000, 64 
dataset = wds.WebDataset(urls).decode("pil").shuffle(5000).batched(batch_size, partial=False) 
loader = wds.WebLoader(dataset, num_workers=4) loader = loader.ddp_equalize(dataset_size // batch_size)
</code></pre>
<p>Could someone please help me understand what is happening in these two pieces of code and how they are different. In the second case is the <code>dataset_size</code> just a nominal size? Which if any is better? I would also appreciate if someone has an example of what is the best way to use Webdataset with pytorch lightning in multi-gpu and multi-node scenario.</p>

H-Huang: <p>Hi, this is for the PyTorch forums so I don’t think many members will have experience / understand WebDataset in detail. You might get more insight posting on that repo’s github issues. However, if there is any issues within the <code>torch.distributed</code> package then please post here.</p>
Loss.float().backward() raises "Found dtype Double but expected Float"

Omroth: <p>I’m explicitly converting my loss (which comes from nn.MSELoss) to type float in this call:</p>
<p><code>loss.float().backward()</code></p>
<p>But I’m still getting this error:</p>
<p><code>Found dtype Double but expected Float</code></p>
<p>For completeness:</p>
<pre><code class="lang-auto">loss.float().backward()
  File "/home/ian/anaconda3/lib/python3.7/site-packages/torch/_tensor.py", line 363, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/home/ian/anaconda3/lib/python3.7/site-packages/torch/autograd/__init__.py", line 175, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
RuntimeError: Found dtype Double but expected Float
</code></pre>
<p>What am I doing wrong?</p>

Omroth: <p>When declaring my Linear layers I needed to cast them to double()</p>
3D Global average pooling to linear

Omroth: <p>Morning,</p>
<p>I have the end result of the 3D convolutional part of my network,  with shape:</p>
<p>n_samples, n_channels, height, width, depth  (in this example this is 1,32, 25, 32, 32)</p>
<p>I wish to:</p>
<ol>
<li>Global average pool this</li>
<li>Concatenate it with a linear extra input (which is of shape n_samples, n_extra_inputs)</li>
<li>Feed it to a fully connected layer</li>
</ol>
<p>The issue I have is in flattening/squeezing the output of the AvgPool3d layer and then adding it to the extra input.  I tried:</p>
<pre><code class="lang-auto">self.global_pool = nn.AvgPool3d(kernel_size=1)
...
x = self.global_pool(x)

torch.cat([x.squeeze, extra_input])
</code></pre>
<p>but x.squeeze just goes from shape [1, 32, 25, 32, 32] to [32, 25, 32, 32] which is clearly not what I need.  This may be a time for View or flatten but rather than trial and error I thought I’d ask the recommended approach.</p>
<p>Thanks,<br>
Ian</p>

J_Johnson: <p>Your kernel_size=1. That defeats the purpose of an AvgPool3d layer as it will just return the exact same as the input. If you want to average each sample to a size of 1, try <code>AdaptiveAvgPool3d(output_size=1)</code>.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool3d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool3d.html</a></p>

Omroth: <p>That’s very helpful, thank you.</p>
<p>So I get a shape 1,32,1,1,1 from that, then squeeze goes to [32].  I then used reshape(x, [1,32]) to get what I needed.</p>
<p>However now I’m struggling with working out how to concatenate the extra data for each sample.  Since my n_samples at the moment is 1 it’s not so complicated but it needs to generalise.</p>
<p>How do I concatenate tensors of shape [1,32] and [1,7] to correctly have a tensor of shape [1,39] ?</p>

Omroth: <p>I see, I need torch.cat(list, axis=1)</p>
Custom endpoint URL for TorchServe

grue: <p>Hi there, I’m trying to deploy a model that must be served at localhost:8080/invocations but TorchServe only seems to allow for localhost:8080/predictions/&lt;model_name&gt;. I can’t see any documentation or examples of this anywhere. You can specify the inference_address in config.properties but can only specify the port not the entire url. Is this possible, or will I have to serve using something like Flask? Thank you</p>
Torchvision.utils.save_image cannot save the whole image

Taejune: <p>Hi,when I use <code>torchvision.utils.save_image</code> for tensor visualization,<br>
it often only saves the upper region of the image.<br>
For instance:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/981cf0b54ee1f4fd44d8a9025f9ea43dc42da7a9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/981cf0b54ee1f4fd44d8a9025f9ea43dc42da7a9" title="스크린샷 2023-02-20 오후 5.29.21"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/981cf0b54ee1f4fd44d8a9025f9ea43dc42da7a9_2_690x233.jpeg" alt="스크린샷 2023-02-20 오후 5.29.21" data-base62-sha1="lHEzWx4J2WbEvrI7ciuXA6Pczr3" width="690" height="233" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/981cf0b54ee1f4fd44d8a9025f9ea43dc42da7a9_2_690x233.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/981cf0b54ee1f4fd44d8a9025f9ea43dc42da7a9_2_1035x349.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/981cf0b54ee1f4fd44d8a9025f9ea43dc42da7a9_2_1380x466.jpeg 2x" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2023-02-20 오후 5.29.21</span><span class="informations">1920×650 66.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have tried saving image only on the main process, and also have tried <code>dist.barrier</code> after the saving code.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/8/98be9478b632d5f0e41a4c3e5aab946d31cd269e.png" data-download-href="https://discuss.pytorch.org/uploads/default/98be9478b632d5f0e41a4c3e5aab946d31cd269e" title="스크린샷 2023-02-20 오후 5.31.17"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98be9478b632d5f0e41a4c3e5aab946d31cd269e_2_690x72.png" alt="스크린샷 2023-02-20 오후 5.31.17" data-base62-sha1="lNeTfMVh6EuLRvSr8I35fdtS9YO" width="690" height="72" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98be9478b632d5f0e41a4c3e5aab946d31cd269e_2_690x72.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/8/98be9478b632d5f0e41a4c3e5aab946d31cd269e_2_1035x108.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/8/98be9478b632d5f0e41a4c3e5aab946d31cd269e.png 2x" data-dominant-color="292A29"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">스크린샷 2023-02-20 오후 5.31.17</span><span class="informations">1368×144 22.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
What do you think is the main problem??</p>
<p>Thanks for reading!</p>
How to Backprop through the training process of a model?

Raffael_Schon: <p>I am using two models in a concatenated fashion. The setup is the following, I have a set of Inputs X1, I am using the first model in order to generate for each X1_i a Y1_i. Based on these X1,Y1 pairs I am then training the second model for E epochs. I have then another dataset X2 X Y2  on which I calculate some loss with the second model. My goal is now to backprop that loss through the training process of the second model in order to update the parameters of the first model which generated the data with which the second model was trained. I have so far running code that looks like somewhat like this:</p>
<pre><code class="lang-auto">optimizer1 = torch.optim.AdamW(model1.params(), lr=0.001)
optimizer2 = torch.optim.AdamW(model2.params(), lr=0.001)

for x_1 in X1:
     y_1 = model1(x_1)
     Y1 = torch.cat(Y1,y_1)

for e in range(E):
     loss_model2 = torch.tensor(0)
     for data_index in range(X1.shape[1]):
          loss_model2 = loss_model2  + someLoss2(model2(X1[data_index ]),Y1[data_index])
    
     loss_model2 .backward(retain_graph=True)
     optimizer2 .step()

loss_model1 = torch.tensor(0)
for ext_data_index in range(X2.shape[1]):
     loss_model1 = loss_model1 + someLoss1(model2(X2[ext_data_index ]),Y2[ext_data_index ])
loss_model1.backward()
optimizer1.step() 
</code></pre>
<p>Sorry for only being able to share pseudo-code since I am not allowed to share company code.<br>
The setup presented here runs without any errors my fear however is that I am not backproping through all trainings iteration of model2 and that I am only backproping through the last iteration and the computation graphs of the previous iterations are discarded</p>
Include 'unlabeled' data with Temporal-Fusion Transformer?

zennsunni: <p>Greetings.</p>
<p>I am experimenting with the Temporal-Fusion Transformer in the forecasting library and have a question about including data without targets. I have a tremendous amount of data with most or all of the the covariate information, but no target value; an analogy would be something like I have a tremendous amount of individual historical stock values, but no stock Index value. I then have a modest dataset of time series that do have target values, e.g. something analogous to a stock index value.</p>
<p>Can these ‘un-labeled’ time-series be included in the training set? My understanding of the TFT is very poor, but my goal is to accomplish something akin to MAE pre-training. Sorry if this is a nonsense question in this context, still trying to wrap my head around how the model works.</p>
Import torch gives libcudnn OSError

Star_Cloud: <p>I have torch 1.8.0 installed. When I try to import it in python 3.8, it gives me the following error</p>
<pre><code class="lang-auto">import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/n/home00/&lt;usr_name&gt;/miniconda3/envs/mae/lib/python3.8/site-packages/torch/__init__.py", line 217, in &lt;module&gt;
    _load_global_deps()
  File "/n/home00/&lt;usr_name&gt;/miniconda3/envs/mae/lib/python3.8/site-packages/torch/__init__.py", line 177, in _load_global_deps
    raise err
  File "/n/home00/&lt;usr_name&gt;/miniconda3/envs/mae/lib/python3.8/site-packages/torch/__init__.py", line 172, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/n/home00/&lt;usr_name&gt;/miniconda3/envs/mae/lib/python3.8/ctypes/__init__.py", line 373, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcudnn.so.8: cannot open shared object file: No such file or directory
</code></pre>

ptrblck: <p>Could you update PyTorch to the latest stable or nightly binary, as <code>1.8.0</code> is quite old by now?</p>
Cant perform forward pass twice in DDP

ashwin2k: <p>I am trying to forward pass 2 different inputs with the same model as shown below:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
        dataloader.sampler.set_epoch(epoch)
        for batch_index, (real,_) in enumerate(dataloader):
            
            disc.zero_grad()
            
            real=real.to(rank)
            noise=torch.randn((batch_size,z_dim,1,1)).to(rank)
            fake_img=gen(noise)
            fake_img_clone=fake_img.detach().clone()
            disc_real=disc(real).reshape(-1)
            lossD_real=critereon(disc_real,torch.ones_like(disc_real))
            
            disc_fake=disc(fake_img.detach()).reshape(-1)
            lossD_fake=critereon(disc_fake,torch.zeros_like(disc_fake))
            lossD = (lossD_fake+lossD_real)/2
                        
            opt_disc.step()
</code></pre>
<p>However, I keep getting the error “one of the variables needed for gradient computation has been modified by an inplace operation”<br>
Setting torch.autograd.set_detect_anomaly(True, check_nan=True) shows that the error occurs in  disc_real=disc(real).reshape(-1), but when I manually debug it, the error occurs only when I add the second forward pass line disc_fake=disc(fake_img.detach()).reshape(-1)</p>
<p>I am currently using the latest version of pytorch. Please help me solve this <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>You are never calling <code>backward</code> in the posted code snippet so I guess you removed important parts of the code. Could you revisit it, please?</p>
How to get npy dataloader

ykeivn: <p>I have a few of .npy files as (25,512,512), and I need use it to input my network by dataloader . What should I do? Please!</p>

ptrblck: <p>You could write a custom <code>Dataset</code> to lazily load each numpy file:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, np_file_paths):
        self.files = np_file_paths
    
    def __getitem__(self, index):
        x = np.load(self.files[index])
        x = torch.from_numpy(x).float()
        return x
    
    def __len__(self):
        return len(self.files)
</code></pre>
<p>After creating the <code>Dataset</code> instance, you could wrap it in a <code>DataLoader</code>, which will create a batches of <code>[batch_size, 25, 512, 512]</code>.</p>

xiew: <p>Hi, I have kind of like a this npy files. I wrote like this:</p>
<pre><code class="lang-auto">    def __init__(self, dir_x, dir_y, transform=None):
        self.transform = transform
        self.datax = np.load(dir_x)
        self.datax = self.datax / 255.0
        self.datax = self.datax.reshape(-1, 1, 64, 64)
        self.datax = torch.from_numpy(self.datax).to(torch.float32)
        
        self.datay = np.load(dir_y)
        self.datay = self.datay / 255.0
        self.datay = self.datay.reshape(-1, 1)

    def __len__(self):
        return self.datax.shape[0]

    def __getitem__(self, idx):
        X = self.datax[idx]
        y = self.datay[idx]
        if self.transform:
            X = self.transform(X)
        return X, y
</code></pre>
<p>but when I train it, I got 0 losses. I don’t understand the problem.</p>

ptrblck: <p>“0 losses” would indicate that your model is able to perfectly fit the training data. As a quick check you can compare the predictions vs. targets and see if that’s indeed the case.</p>
How to conver a FloatTensor to LongTensor?

luodahei: <p>i have try tensor.long() but can not conver it<br>
thanks</p>

iariav: <p>.type(torch.LongTensor)</p>

justusschock: <p>What error occurs if trying to convert it with <code>tensor.long()</code>? This should usually work.</p>

klory: <p>Yes, I also wonder when it does not work…</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/22eb5f60c1340ab67cabcdc2ea6c06ccb024740f.PNG" alt="Capture" data-base62-sha1="4YUw4T8Z3lJyKbQAKAdut3feN3N" width="333" height="117"></p>

andreiliphd: <p>I use X_train_torch = torch.from_numpy(X_train).long() and it’s not working.<br>
I tried to use also .type(torch.LongTensor) but it doesn’t work anyway.<br>
RuntimeError: Expected object of type torch.LongTensor but found type torch.FloatTensor for argument <span class="hashtag">#2</span> ‘mat2’<br>
What’s the problem?</p>

andreiliphd: <p>I am broke or PyTorch broke. Whatever I do error is the same. The problem with type. Whatever I cast no result, the same error. What ever I do the same. I tried to close and open Jupyter Notebook, copied it. Error is the same. I have dying Li problem.</p>

justusschock: <p>From looking at the actual message, I think the problem is not your <code>X_train_torch</code> but the tensor you try add/subtract/… to/from/with it since they need to have the same type. Can you post a small snippet to reproduce the error?</p>

andreiliphd: <p>Thank you very much for help.</p>
<pre><code class="lang-auto">loss_fn = torch.nn.CrossEntropyLoss()
learning_rate = 0.01
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
for t in range(5000):
    y_pred = model(X_train_torch)
    loss = loss_fn(y_pred, y_train_torch)
    print(t, loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()```</code></pre>

andreiliphd: <p>I moved forward. But thank you <a href="https://discuss.pytorch.org/u/justusschock">justusschock</a> for your response. I changed the structure on my neural network and the problem disappeared.</p>

Tasnim_Dardouri: <p>tensor.long() doesn’t change the type of tensor permanently. Instead try: out = tensor.long() then use out as it’s type is LongTensor.</p>

Kool_Cool: <pre><code class="lang-auto">x2 = torch.ones(3, 2)



x2 = (torch.ones(3, 2)).long()

</code></pre>
<p>I used  this long</p>
Install Pytorch GPU with pre-installed CUDA and cudnn

weiz: <p>As the title suggests, I have pre-installed CUDA and cudnn (my Tensorflow is using them).</p>
<p>The version of CUDA is  <strong>10.0</strong>  from <code> nvcc --version</code> .</p>
<p>The versiuon of cudnn is  <strong>7.4</strong> .</p>
<p>I am trying to install pytorch in a conda environment using  <code>conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</code> .</p>
<p>However, the installed pytorch does not detect my GPU successfully.</p>
<p>Does anyone know if there is a way to install GPU-version pytorch with a specific CUDA and cudnn version? I do not want to change CUDA and cudnn version because my Tensorflow is using them.</p>
<p>Any ideas would be appreciated!</p>

ptrblck: <p>Your system installations of CUDA and cudnn won’t be used, if you install PyTorch binaries with these libs.<br>
E.g. <code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</code> will install CUDA10.1 and cudnn in your current conda environment.</p>
<p>What kind of error message are you seeing and which GPU are you using?</p>

weiz: <p>I didn’t see any error message. I just couldn’t use my GPU, meaning that <code>print(torch.cuda.is_available())</code> gives me False. My GPU is 2080 Ti using 410.xx driver.</p>
<p>If I install by <code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch will install CUDA10.1</code>, does that mean I will use CUDA 10.1 in the conda environment instead of the one in the system?</p>

ptrblck: <p>Yes, that would use the shipped CUDA10.1 version from the binaries instead of your local installation.<br>
If you want to use the local CUDA and cudnn, you would need to build from source.</p>
<p>EDIT:<br>
Note that CUDA10.0.130 needs driver 410.48 as described <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver">here</a>.</p>

weiz: <p>I just checked my GPU driver version, which has no issue.</p>
<p>Regarding the first paragraph of your reply, does that mean I should have no problem if I install CUDA through conda and use the shipped CUDA 10.1? However, this is not the case for me since my pytorch is still not detecting my GPU even if I install the whole package from conda.</p>

weiz: <p>So I think I figure it out. It turns out that it is because I didn’t reboot the system after installing pytorch using conda.</p>
How to collect libtorch package like the official release when building from source?

lifengcai: <p>I am trying to bulid libtorch from source by <a href="https://github.com/pytorch/pytorch/blob/master/docs/libtorch.rst" rel="nofollow noopener">this</a>.<br>
I found that build/include directory only contain <code>c10d(dir)</code> and <code>sleef.h(file)</code>, but official pre-build release contain <code>(ATen c10 caffe2 pybind11 TH THC THCUNN THNN torch) dirs</code>.</p>
<p>so, how can we get libtorch package just like official pre-build release(libtorch-win-shared-with-deps-debug-latest), particually <strong>windows</strong> platform.</p>

tom: <p>I think using an appropriate CMake + make install should work, e.g. the Android build does this. You want to disable the Python bit for this.<br>
The suggested alternative there works well, too - building Python and picking the lib and include. Libtorch 1.0 used to be built that way (actually extracting from the whl…).</p>
<p>Best regards</p>
<p>Thomas</p>

Virajika: <p>I found this <a href="https://github.com/pytorch/builder/blob/main/manywheel/build_libtorch.sh" rel="noopener nofollow ugc">script</a>, is there any other way to create the LibTorch package the same as release?</p>
Dataset and training of pre-trained models in pytorch vision

zaratruta: <p>I’m using te pre-trained models available here: <a href="https://pytorch.org/vision/stable/models.html" class="inline-onebox" rel="noopener nofollow ugc">Models and pre-trained weights — Torchvision main documentation</a></p>
<p>I would like to be sure about the dataset used for training those models (for example, imagenet with 1k categories) and he training procedure (for example, if images are normalized to a standard size before training, etc).</p>
<p>Where can I find this information?</p>

ConvolutionalAtom: <p>Two relevant sources on this matter:</p>
<ol>
<li>
<a href="https://pytorch.org/vision/stable/training_references.html" rel="noopener nofollow ugc">https://pytorch.org/vision/stable/training_references.html</a>, training scripts references</li>
<li>
<a href="https://pytorch.org/vision/stable/models/generated/torchvision.models.resnet50.html#torchvision.models.resnet50" class="inline-onebox" rel="noopener nofollow ugc">resnet50 — Torchvision main documentation</a>, where you can see preprocessing steps. The same type of info is available for other models</li>
</ol>
<p>Using the latter, I was able to reproduce some well-known finetuning scores based on resnet50 efficiennet L, etc</p>
Purpose of the function "Embedding_Dense_Backward" Defined in File Functions.h

Yash_Singhal: <p>I am a Research Student doing profiling on a pytorch code related to NLP. During Profiling I realized that a certain function “embedding_dense_backward” does have slightly more Cpu time than Cuda time. I went into the documentation <a href="https://pytorch.org/cppdocs/api/function_namespaceat_1aa5f93736b786be2be66464abc53de431.html" rel="noopener nofollow ugc">here</a><br>
however it wasn’t helpful. Thus could anyone tell what’s this function really do, to my understanding it might be related to updation on an embedding vector during backpropogation. Apologies if question is trivial</p>
RuntimeError: Cannot find ptxas

vgoklani: <p>I’m using the NVIDIA Docker container and the latest pytorch nightly:</p>
<pre><code class="lang-auto">nvcr.io/nvidia/cuda:11.8.0-devel-ubuntu22.04

pip3 install --pre torch torchvision torchaudio --force-reinstall --index-url https://download.pytorch.org/whl/nightly/cu118

torch.__version__ == 2.0.0.dev20230218+cu118

</code></pre>
<p>I can no longer compile after updating to the latest pytorch nightly.</p>
<p>This is the error:</p>
<p><code>RuntimeError: Cannot find ptxas</code></p>
<p>I saw this on twitter:</p>
<aside class="onebox twitterstatus" data-onebox-src="https://twitter.com/ptrblck_de/status/1626710976342163458">
  <header class="source">

      <a href="https://twitter.com/ptrblck_de/status/1626710976342163458" target="_blank" rel="noopener nofollow ugc">twitter.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://pbs.twimg.com/profile_images/732154307173322752/YM1VhUkp_400x400.jpg" class="thumbnail onebox-avatar" width="300" height="300">
<h4><a href="https://twitter.com/ptrblck_de/status/1626710976342163458" target="_blank" rel="noopener nofollow ugc">ptrblck</a></h4>
<div class="twitter-screen-name"><a href="https://twitter.com/ptrblck_de/status/1626710976342163458" target="_blank" rel="noopener nofollow ugc">@ptrblck_de</a></div>

<div class="tweet">
  <span class="tweet-description"><a href="https://twitter.com/deliprao" target="_blank" rel="noopener nofollow ugc">@deliprao</a> The PyTorch binaries ship with their own CUDA dependencies, so you would also need to install a proper NVIDIA driver (not the full CUDA toolkit). `torch.compile` uses `ptxas` for its code-gen, and this binary should also ship in the current nightlies now.</span>
</div>

<div class="date">
  <a href="https://twitter.com/ptrblck_de/status/1626710976342163458" class="timestamp" target="_blank" rel="noopener nofollow ugc">10:31 PM - 17 Feb 2023</a>

    <span class="like">
      <svg viewbox="0 0 512 512" width="14px" height="16px" aria-hidden="true">
        <path d="M462.3 62.6C407.5 15.9 326 24.3 275.7 76.2L256 96.5l-19.7-20.3C186.1 24.3 104.5 15.9 49.7 62.6c-62.8 53.6-66.1 149.8-9.9 207.9l193.5 199.8c12.5 12.9 32.8 12.9 45.3 0l193.5-199.8c56.3-58.1 53-154.3-9.8-207.9z"></path>
      </svg>
      32
    </span>

</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>so presumably this is just a config/path error? How should I go about fixing this?</p>
<p>Thanks!</p>

ptrblck: <p>Our latest rebuild as of yesterday should have fixed it, but let me double check it.</p>
<p>EDIT: Something still seems to be broken in the latest build. We are tracking the issue <a href="https://github.com/pytorch/builder/issues/1318">here</a>.</p>
GPU Kernels for Block-Sparse matmul and conv

Carl: <p>Hello,</p>
<p>OpenAI has released GPU kernels for block-sparse operations, and wrappers in TensorFlow.</p>
<p><a href="https://github.com/openai/blocksparse" rel="nofollow noopener">https://github.com/openai/blocksparse</a></p>
<p>Looks like something not to complicated to port to PyTorch. I’ve never played with GPU kernels, never contributed to PyTorch. Anybody interested to do it? I might do it eventually if nobody’s interested.</p>
<p>Cheers!</p>

dylan: <p>Curious if block-sparse has been implemented?</p>
Pytorch rocm install error. How to fix this?

Vic_V: <p>I have installed pytorch in a way recommended on <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a> using this command:</p>
<pre><code class="lang-auto">
sudo pip3 install torch torchvision --extra-index-url https://download.pytorch.org/whl/rocm5.2

</code></pre>
<p>But after installation, I run a python script with pytorch and get error:</p>
<pre><code class="lang-auto">/home/vic/.local/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libc10_cuda.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
</code></pre>
<p>How to fix that?</p>
<p>torch version:</p>
<pre><code class="lang-auto">import torch
torch.__version__

'1.13.1+rocm5.2'
</code></pre>
<p>torchvision version(error):</p>
<pre><code class="lang-auto">import torchvision
/home/vic/.local/lib/python3.10/site-packages/torchvision/io/image.py:13: UserWarning: Failed to load image Python extension: libc10_cuda.so: cannot open shared object file: No such file or directory
  warn(f"Failed to load image Python extension: {e}")
</code></pre>
<p>python version is 3.10<br>
and that is Arch linux.</p>

Vic_V: <p>Remove all torch related directories from  /home/vic/.local/lib/python3.10/site-packages/ and reinstall torch and everything is fixes. Sorry for the disturbance.</p>
Error in PyTorch program

PyTorch2603: <p>Hi. In my code I have a list x that I have to convert to a tensor. However, when I do:<br>
x=torch.tensor(x)<br>
I get the following warning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
I saw this same error online being asked before but for some reason I’m unable to change the code so the warning goes away. What code would I have to substitute instead of x=torch.tensor(x) to resolve this warning? Thanks!</p>

ptrblck: <p>The warning is raised if <code>x</code> is already a tensor as seen here:</p>
<pre><code class="lang-python">x = torch.randn(10)
x = torch.tensor(x)
# UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).

x = [0, 1, 2, 3]
x = torch.tensor(x) # no warning
</code></pre>
<p>In this case you won’t need to pass it again into <code>torch.tensor</code> and could <code>clone</code> it instead if needed.</p>
Import torch fails - duplicate CUDA device?

fxgovers: <p>Pytorch keeps failing. Error message below when I run “import torch”</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
[F …\c10\core\CopyBytes.cpp:24] Check failed: g_copy_bytes[0][from][to] == nullptr &amp;&amp; g_copy_bytes[1][from][to] == nullptr Duplicate registration for device type pair CUDA, CUDA</p>
</blockquote>
</blockquote>
</blockquote>
<p>I have the following installed on top of Python 3.8.5</p>
<p>torch                   1.8.1+cu111<br>
torchaudio              0.8.1<br>
torchvision             0.9.1+cu102</p>
<p>I note that Torchvision does not seem to be compatible with CUDA 11.1.  There is not another verson of torchvision available.  Suggestions?</p>
<p>Note that after this error, the python interpreter crashes.</p>
<p>Running on windows 10 under Anaconda<br>
Thanks<br>
Francis</p>

MaarufB: <p>Hello bro. Have you fixed that? I am currently encountering that issue also.</p>

Wang_Zhiqiang: <p>Same problem here, have no idea what to do</p>

ptrblck: <p>PyTorch 1.8.1 is quite old and not supported anymore, so could you update to the latest stable or nightly release and post a minimal, executable code snippet to reproduce the issue in case it still fails?</p>
Layer-wise linear combination of two models

Hannes_F: <p>I have two trained neural network models M1 and M2 in Pytorch.</p>
<p>Both have exactly the same neural network architecture (same # of layers, which we will denote by ‘n’). Furthermore, I have a n-dimensional vector ‘lambda’, where each value ‘lambda_i’ is in range [0, 1].</p>
<p>What I want to calculate now is the ‘layer-wise’ linear combination ‘M = lambda * M1 + (1 - lambda) * M2’. Means that the i-th layer in M is calculated as the linear combination of the i-th layer in M1 (using factor ‘lambda_n’) and the i-th layer in M2 (using factor ‘1 - lambda_i’). The linear combination shall be done both for ‘weight’ and ‘bias’ of the i-th layer.</p>
<p>How can I calculate this in Pytorch ?</p>

ptrblck: <p>You could iterate both <code>state_dict</code>s of the models, create new parameters by applying your weighting, and create a new <code>state_dict</code>. Afterwards, you could load it into a model object.</p>

Hannes_F: <p>Thank you !</p>
<p>How can I determine which keys of the ‘state_dict’ dictionary belong to the <em>same</em> layer ?</p>
<p>I suppose for each key its layer name is the first part of the key name, before the <em>last</em> occurence of the dot character “.” in the key name string.<br>
Is that right ?</p>
<p>As in the example at <a href="https://brsoff.github.io/tutorials/beginner/saving_loading_models.html" class="inline-onebox" rel="noopener nofollow ugc">Saving and Loading Models — PyTorch Tutorials 1.0.0.dev20181128 documentation</a><br>
and in <a href="https://discuss.pytorch.org/t/how-do-i-update-old-key-name-in-state-dict-to-new-version/76026" class="inline-onebox">How do I update old key name in state_dict to new version</a></p>
<p>Btw, nice greetings to California. I hope GTC 2024 is physically again, looking forward to present again at this great conference.</p>

ptrblck: <p>Each <code>key</code> of the <code>state_dict</code> will be unique and represents a registered parameter or buffer. Since both models have the exact same architecture you should be able to iterate one <code>state_dict</code> and use its <code>key</code> to index both dicts inside the loop.</p>
<p>Greetings back!</p>
Calculation in forward based on the value of inputs (Xs)

amirmob2000: <p>Hi there,<br>
How  is it possible to perform the following conditional calculations in a vectorized form?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class ExampleMultinet(nn.Module):
    def __init__(self):
 
        super().__init__()
        self.n_features=10-1 # first column of input is used for conditioning
        self.net1 = nn.Sequential(nn.Linear(9,5),nn.ReLU(),nn.Linear(5,3))
        self.net2 = nn.Sequential(nn.Linear(9,5),nn.ReLU(),nn.Linear(5,3))
        self.net3 = nn.Sequential(nn.Linear(9,5),nn.ReLU(),nn.Linear(5,3))

    def forward(self, X):
        # X dim is (-1,10)
        out=[]
        for x in X:
            if x[0].item()&lt;8:
                out.append(self.net1(x[1:].reshape(-1,self.n_features)))
            elif (8&lt;=x[0].item()&lt;20):
                out.append(self.net2(x[1:].reshape(-1,self.n_features)))
            elif x[0].item()&gt;=20:
                out.append(self.net3(x[1:].reshape(-1,self.n_features)))
        
        return torch.cat(out,dim=0)

model=ExampleMultinet()
x=torch.randn(4,10)*20
y=model(x)
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/amirmob2000">@amirmob2000</a>,</p>
<p>Have a look at <code>functorch.vmap</code>, docs here: <a href="https://pytorch.org/functorch/stable/generated/functorch.vmap.html" class="inline-onebox" rel="noopener nofollow ugc">functorch.vmap — functorch 1.13 documentation</a></p>
<p>It allows for vectorization of pytorch operations and should work in your case!</p>
Named_modules can't get all the layer

MartinZhang: <p>Hi, Today I try use the named_modules to get the all layer from the net</p>
<p>like this:</p>
<pre><code class="lang-auto">for name, layer in net.named_module():
    print(name, layer)
</code></pre>
<p>this can’t print the all layer ?</p>

ptrblck: <p><code>.named_module</code> is not a valid method and you should use <code>.named_modules</code> instead which will print all registered modules as seen here:</p>
<pre><code class="lang-python">model = models.resnet18()
for name, layer in model.named_modules():
    print(name, layer)
</code></pre>
Custom BatchSampler for two-step mini-batch

Saeed_Izadi: <p>Hi All,</p>
<p>In the data preparation phase for my network, I read an image one at a time, and then, I want to extract several patches from this image as my mini-batch. In other words, the data preparation consists of two steps: 1) read an image and 2) extract random patches to form the mini-match.</p>
<p>What’s the proper way to use BatchSampler to implement this?</p>
<p>Thanks,<br>
Saeed</p>

ptrblck: <p>You could just sample in <code>__getitem__</code> and stack the patches into the batch dimension:<br>
Here is a small example sampling <code>5x5</code> patches. These patches are returned as a 4-dimensional <code>tensor</code> from <code>__getitem__</code>. During the training you could push these patches into the batch dimension with <code>view</code>.</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self):
        self.data = torch.randn(100, 3, 24, 24)
        
    def __getitem__(self, index):
        # Get current image
        image = self.data[index]
        # Sample patches
        patches = self.sample_pathes(image)
        
        return patches
    
    def sample_pathes(self, image):
        # Your sampling logic
        size = 5
        patches = []
        for i in range(5):
            patch = image[:, i:i+size, i:i+size]
            patches.append(patch)
        patches = torch.stack(patches)
        
        return patches
    
    def __len__(self):
        return len(self.data)


dataset = MyDataset()

loader = DataLoader(
    dataset,
    batch_size=10,
    shuffle=False,
    num_workers=2
)

loader_iter = iter(loader)
x = loader_iter.next()
x = x.view(-1, 3, 5, 5)
</code></pre>

Jaideep_Valani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> i have this requirement<br>
Out of Big  N samples I want to randomly select M samples every epoch for a batch. No weighted criteria<br>
How can we do that ?</p>

ptrblck: <p>If you want to create batches from a subset <code>M</code> of the original dataset with <code>N</code> samples, you could use <code>torch.utils.data.Subset</code> and pass the desired indices to this class.</p>

Jaideep_Valani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  I need to randomly select M samples every epoch.so M should always get changed.<br>
Say I dataframe Df =1M  rows<br>
I want to select 500k rows randomly all the time.<br>
Do I need some thing like this ,that is my own sampler</p>
<pre><code class="lang-auto">class YourSampler(Sampler):
    def __init__(self, mask):
        self.mask = mask

    def __iter__(self):
        return (self.indices[i] for i in torch.nonzero(self.mask))

    def __len__(self):
        return len(self.mask)
</code></pre>

ptrblck: <p>Couldn’t you recreate the <code>Subset</code> in each epoch and pass it to a new <code>DataLoader</code> again?</p>

Jaideep_Valani: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I use now Weighted Random Sampler in desired ration with No of samples Drawn as Half . which is M.<br>
I assume every epoch WRS will draw new pool of data  as per Ratio given</p>