akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Did torch.sort drastically improve in 1.10?

oleitersdorf: <p>Hi,</p>
<p>Did <code>torch.sort</code> on CUDA for large arrays <em>drastically</em> improve from version 1.9 to 1.10? The release notes of 1.10 don’t mention any major improvement to sorting.</p>
<p>I ask as the below test for sorting 10^8 floating-point numbers results in a throughput of <em>85 MKPS</em> (million keys sorted per second) on torch 1.9 and <em>1585 MKPS</em> on torch 1.10. That’s a massive difference. Note that I’m running the same script on the same GPU (GeForce RTX 3070), and only changing the PyTorch version.</p>
<pre><code class="lang-auto">import time
import torch

n = 1 * (10 ** 8)
x = torch.randn((n, ), device='cuda')

# Start time measurement
torch.cuda.synchronize()
t = time.perf_counter()

output, _ = torch.sort(x)

# End time measurement
torch.cuda.synchronize()
sort_time = time.perf_counter() - t

print(f'Latency: {sort_time}')
print(f'Throughput: {round((n / sort_time) / 1e6, 3)} MKPS')
</code></pre>
<p>A few comments:</p>
<ol>
<li>
<p>I suspected that lazy evaluation may be involved. To test this idea, I added <code>print(torch.diff(output, 1).prod())</code> after the sort (and before the time measurement), a value that can only be computed if the array is truly sorted. The results remained roughly the same (1489 MKPS).</p>
</li>
<li>
<p>I’ve also reproduced these results on a GTX 2080 Ti, roughly the same values as the 3070.</p>
</li>
<li>
<p>I’ve also run the above script in a for loop and averaged the results (e.g., to avoid issues with warmup and noise).</p>
</li>
</ol>
<p>Was the sorting mechanism indeed drastically improved, or am I missing something in this phenomenon?</p>

ngimel: <p>1.9 had a perf bug that made sorting 1d tensors or multi-d tensors with few sorted slices very slow, this bug was fixed for 1.10 (along with minor perf improvement brought by new cuda version).</p>

oleitersdorf: <p>Interesting. Can you point me to the pull request for the bug?</p>
<p>By the way, what sorting algorithm does PyTorch implement for this scale?</p>

ngimel: <p>Here’s a fix <a href="https://github.com/pytorch/pytorch/pull/64127" class="inline-onebox" rel="noopener nofollow ugc">Bring back old algorithm for sorting on small number of segments by zasdfgbnm · Pull Request #64127 · pytorch/pytorch · GitHub</a>, it has a link to the issue describing regression and the issue has a link to original PR introducing it.<br>
Pytorch is calling into cub for almost all (and definitely all large) inputs.</p>
Why my trained model output is same for each random input?

Nourollah.am: <p>I trained my model on the Python platform. after training, I faced up with same output for each random input. I solved this problem by deactivating BatchNorm layers with the <code>model.eval()</code> method. but when I tried to load my trained model in C++ with <a href="https://medium.com/@albertsundjaja/deploying-trained-pytorch-model-in-c-2f61f46bacfe" rel="noopener nofollow ugc">Pytorch C++ API</a>, this problem showed up again, and <code>model.eval()</code> not helping me at this time. I faced the same output for each random input again.</p>
<p>This is my C++ model loading function:</p>
<pre><code class="lang-auto">std::vector&lt;torch::jit::script::Module&gt; module_loader(std::string file_addr) {
    std::vector&lt;torch::jit::script::Module&gt; modul;
    torch::jit::script::Module model = torch::jit::load(file_addr);
    model.eval();
    modul.push_back(model);
    return modul;
}
</code></pre>
<p>And this is my testing function:</p>
<pre><code class="lang-auto">void test(std::vector&lt;torch::jit::script::Module&gt; &amp;model) {
    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(torch::rand({1, 2, 64, 172}));
    torch::Tensor output = model[0].forward(inputs).toTensor();
    std::cout &lt;&lt; output &lt;&lt; std::endl;
}
</code></pre>
<p>After all, I put it all together in main() like this:</p>
<pre><code class="lang-auto">int main() {
    auto modul = module_loader(MODEL_ADDRESS);
    test(modul);
}
</code></pre>
<p>MODEL_ADDRESS is a macro for the address of the trained model on my local disk.</p>
<p>The output of the program is this for every run:</p>
<blockquote>
<p>0.3231 [ CPUFloatType{1,1} ]</p>
</blockquote>
Separating integer digits in three digit groups

Kamil_Adamczewski: <p>Is it possible to print an integer in a way that thousands, millions, billions, etc. are separated by commas?<br>
For example:<br>
4,374,125,945</p>

googlebot: <p>you should read “Format Specification Mini-Language” in python docs.</p>
<p>the basic code for this is “,d” (e.g. f"{1000:,d}")</p>
How to use predict function, when model was trained with DataParallel?

scribblepad: <p>Hello,</p>
<p>What is the correct way to get predictions when model is trained with DataParallel?</p>
<p>I’ve trained a model which uses the following to make use of multiple GPUs.<br>
model = nn.DataParallel(model)</p>
<p>I save the model with,<br>
torch.save(model, model_home+‘best_model.pth’)</p>
<p>Load and run predictions:<br>
best_model = torch.load(model_home+‘best_model.pth’)</p>
<p>predictions = best_model.predict(x_tensor)</p>
<p>I run into the below error:<br>
ModuleAttributeError: ‘DataParallel’ object has no attribute ‘predict’</p>
<p>Thanks</p>

ptrblck: <p>In case your original model provides a <code>predict</code> method, you could use <code>best_model.module.predict</code>.<br>
<code>nn.DataParallel</code> will use the <code>forward</code> method to in its data parallel approach and will ignore your custom methods. If you want to use <code>predict</code> in the same data parallel way, you would have to use it in your <code>forward</code> method instead.</p>

scribblepad: <p>I tried to change it this way without the predict function when DataParallel was used while training.</p>
<p>predictions  = best_model(x_tensor)</p>
<p>It seems to be predicting, but not sure if this is the right way?</p>

ptrblck: <p><code>predictions = best_model(x_tensor)</code> would call into <code>__call__</code> and then into the <code>forward</code> method.<br>
I don’t know how <code>predict</code> is defined and what the difference between it and <code>forward</code> would be.<br>
In case both methods are doing the same, your approach should be fine.</p>

scribblepad: <p>Got it. thank you.<br>
My understanding so far, for training without DataParallel, I can use predict function, other wise use the model(tensor) which would use <strong>call</strong> (then the forward method), which will work to get predictions.</p>

ptrblck: <p>This might work, but note that it’s not a general rule since your <code>predict</code> method is a custom function which is not a standard method in <code>nn.Module</code>.<br>
The usual and required workflow is to override the <code>forward</code> method. You should thus compare your <code>predict</code> method with the <code>forward</code> and check what the difference is.<br>
<code>nn.DataParallel</code> doesn’t have any knowledge about custom methods (such as <code>predict</code>) and thus will use the standard <code>forward</code> method to use the data parallel approach.</p>

scribblepad: <p>Got it. That makes sense. Will keep in mind. Thank you</p>
What is the best way to approach a problem, when the data needs to go through two routes in a network given and argument?

gebrahimi: <p>I have a situation where I have 6 stack of decoders on top of each other and after that there are 2 heads and each head is a decoder. Some samples in the batch should go through the first 6 layers and the first head and some samples should go through the first six heads and the second decoder head. given a a batch size, i have a tensor of size batch_size of 1 and 0s which tells me what route the data should take. What is the best way to handle this scenario and train the network?</p>
<p>should i pass all the data through both networks and then slice the outputs based on the output i need or there is a better way to do that without needing to feed the data twice? Can mask variables help here?</p>
<p>Thanks.</p>

ptrblck: <p>Since the entire batch seems to use the first 6 layers, you could directly pass it to them and split the batch afterwards e.g. by indexing. Once you have the two chunks you could then pass them to the corresponding head and if needed concatenate them afterwards (in the same order).</p>
Problem with import timm for using it

prog_asker: <p>I tried to load a model using the interface provided in timm by</p>
<blockquote>
<p>import timm<br>
model = timm.create_model(‘model_name’, pretrained=True)</p>
</blockquote>
<p>but got</p>
<blockquote>
<p>[libprotobuf FATAL google/protobuf/stubs/common.cc:87] This program was compiled against version 3.9.2 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.18.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in “bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc”.)<br>
terminate called after throwing an instance of ‘google::protobuf::FatalException’<br>
what():  This program was compiled against version 3.9.2 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.18.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in “bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc”.)<br>
Aborted (core dumped)</p>
</blockquote>

ptrblck: <p>Are you mixing TF/PyTorch together, as this error seems to be raised by TF?</p>
<pre><code class="lang-auto">Version verification failed in “bazel-out/k8-opt/bin/tensorflow/core/framework/tensor_shape.pb.cc”
</code></pre>

ptrblck: <p>Based on the warning it seems that just no pretrained weights exist for this model and I think you would need to train it from scratch (or check if updating <code>timm</code> might ship with new pretrained weights).</p>
Complex number DataLoader

gokijam: <p>Hi, I’m new to Pytorch.<br>
I want to train Autoencoder and Denoising Autoencoders with my 30k complex number array data.<br>
I saved every single array(1,480,480) as npy.<br>
How should I make a data loader for AE and DAE with this data?<br>
I mean, I tried to put it into one list (yeah list with 30k elements) make a tensor from this, it failed.<br>
My complex number arrays represent images after FFT.<br>
I will be grateful for any clue how to work with this data in PyTorch.</p>

ptrblck: <p>What exactly failed while trying to create the tensor? Could you post the error message and the corresponding code, please?</p>

gokijam: <p>OK, I should use torch.from_numpy instead torch.ComplexType.<br>
Now It works.</p>
Name of layers in repeatative blocks

kasra_sehat: <p>i want to implement a deep network like resnet. i want to prepare a block of conv and linear layers that are repeated through the code. i believe the name of layers will result in issues on backprop and training process because of repeatation of layers names. is it true? if yes, so what is the soloution? how can i create a block which changes layers name after each repeatation?</p>

albanD: <p>Hi,</p>
<p>No naming of your variable will be a problem for autograd.<br>
Autograd works “below” torch.nn so you can set up your network whichever way you want and it will work just fine.</p>