akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Build dataset with multiple class masks for EndoCV 2020

username: mrleo194
cooked: <p>I want to build a dataset for segmentation for EndoCV 2020 dataset. In this dataset, for example, <code>image = abc.jpg</code>, with 2 classes in this image. The corresponding masks are <code>mask_1 = abc_class1.tif</code> and ‘mask_2 = abc_class2.tif’. How can I build a dataset that PyTorch can train?</p>

username: ptrblck
cooked: <p><a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">This tutorial</a> might be a good starter which explains how to write a custom <code>Dataset</code>.</p>
Mat1 and mat2 shapes cannot be multiplied in pytorch lightning

username: Manjuphoenix
cooked: <p>I’m new to pytorch lightning and currently training a custom model to classify diseases which had 2 pairs of cnn, maxpool and relu layers. The same model was working very well when using normal pytorch module. I wanted to add more functionality so choose pytorch lightning. The same architecture is now giving error, can someone please guide me through<br>
Normal pytorch module:<br>
class NeuralNetwork(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.cnn = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=1, padding=2)<br>
self.relu = nn.LeakyReLU()<br>
self.maxpool = nn.MaxPool2d(kernel_size=2)<br>
self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1, padding=2)<br>
self.relu2 = nn.LeakyReLU()<br>
self.maxpool2 = nn.MaxPool2d(kernel_size=2)<br>
self.fc1 = nn.Linear(32<em>65</em>65, 1000)<br>
self.fc2 = nn.Linear(1000, 12)</p>
<pre><code>def forward(self, x):
    out = self.cnn(x)
    out = self.relu(out)
    out = self.maxpool(out)
    out = self.cnn2(out)
    out = self.relu2(out)
    out = self.maxpool2(out)
    out = out.view(out.size(0), -1)
    out = self.fc1(out)
    out = self.fc2(out)
    return out
</code></pre>
<p>Implementing pytorch lightning:<br>
class NeuralNetwork(pl.LightningModule):<br>
def <strong>init</strong>(self):<br>
super(NeuralNetwork, self).<strong>init</strong>()<br>
self.cnn = nn.Conv2d(in_channels=3, out_channels=16, kernel_size=5, stride=1, padding=2)<br>
self.relu = nn.LeakyReLU()<br>
self.maxpool = nn.MaxPool2d(kernel_size=2)<br>
self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=3, stride=1, padding=2)<br>
self.relu2 = nn.LeakyReLU()<br>
self.maxpool2 = nn.MaxPool2d(kernel_size=2)<br>
self.fc1 = nn.Linear(32<em>65</em>65, 1000)<br>
self.fc2 = nn.Linear(1000, 14)<br>
self.loss = nn.CrossEntropyLoss()</p>
<pre><code>def forward(self, x):
    out = self.cnn(x)
    out = self.relu(out)
    out = self.maxpool(out)
    out = self.cnn2(out)
    out = self.relu2(out)
    out = self.maxpool2(out)
    out = out.view(out.size(0), -1)
    out = self.fc1(out)
    out = self.fc2(out)
    return out
</code></pre>

username: ptrblck
cooked: <p>I cannot see any differences in the model definitions and it doesn’t make sense to me that Lightning would raise a shape mismatch error. Assuming you can reproduce the issue with a minimal, executable code snippet I would recommend to create an issue on their <a href="https://github.com/Lightning-AI/lightning/issues">GitHub</a> so the Lightning devs could take a look at it.</p>
What is the proper way to write a multi headed model in C++?

username: Alio
cooked: <p>This model has</p>
<pre><code class="lang-auto">What is the proper way to write a multi headed model in C++?

Does the code below suffice or do I need to some some special module inside the torch::nn library?

std::tuple&lt;torch::Tensor, torch::Tensor&gt; Critic_Net : torch::nn::Module {
    torch::Tensor next_state_batch__sampled_action;
    // This network returns just one value
    public:
    Critic_Net() {
        // Construct and register two Linear submodules.
        lin1 = register_module("lin1", torch::nn::Linear(427, 42));
        lin2 = register_module("lin2", torch::nn::Linear(42, 286));
        head_1 = register_module("head_1", torch::nn::Linear(286, 1));        
        head_2 = register_module("head_2", torch::nn::Linear(286, 1));
        lin1-&gt;to(device);
        lin2-&gt;to(device);
        head_1-&gt;to(device);
        head_2-&gt;to(device);
    }
    torch::Tensor forward(torch::Tensor next_state_batch__sampled_action) {
        auto h = next_state_batch__sampled_action;
        h = torch::relu(lin1-&gt;forward(h));
        h = torch::tanh(lin2-&gt;forward(h));
        h1 = head_1-&gt;forward(h);
        h2 = head_2-&gt;forward(h);
        return {h1, h2};
    }
    torch::nn::Linear lin1{nullptr}, lin2{nullptr}, head_1{nullptr}, head_2{nullptr};
};
</code></pre>
Teacher forcing option with PackedSequence input

username: Messiah
cooked: <p>Hi,</p>
<p>Can a teacher forcing option (i.e. using predicted vs. ground-truth label as current input during training for each time step in a sequence prediction) be controlled when using a PackedSequence object as input to an RNN? It seems like teacher forcing is performed by default in such a case. I am not sure what the process would be like as there are no code examples I could find online. Examples I did find manually iterate through each time step of a sequence instead.</p>

username: vdw
cooked: <p>See my <a href="https://discuss.pytorch.org/t/is-teacher-forcing-default-for-nn-lstm/71379/7">reply</a> to your other post.</p>
Lightning: Why does the dataloader run multiple times and take up all the RAM?

username: SAMOKILLER
cooked: <p>I am trying to run parallel learning with a large dataset.<br>
In my understanding, the dataset should be loaded into RAM, and then the batches are sent to the GPU.<br>
However, for some reason the data is loaded into RAM for each GPU involved in the training.<br>
That is, when training for 6 gpu runs simultaneously 6 datoaders and each in RAM.<br>
As a result, even before the start of training the memory in RAM is already completely exhausted, because the dataset is duplicated there 6 times!<br>
Firstly, it is impossible to load a large dataset for training.<br>
Secondly, it is very long.</p>
<pre><code class="lang-auto">if __name__ == '__main__':
    Pytorch_lightning_MNIST_my = Pytorch_Lightning_my()
    trainer = pl.Trainer(accelerator='gpu', devices=6, max_epochs=EPOCHS, strategy=DDPStrategy(find_unused_parameters=False))
    DS = np.vstack((np.loadtxt(learningDatasetFile, skiprows=1, delimiter=",", dtype=np.float32, max_rows=125000), np.loadtxt(validateDatasetFile, skiprows=1, delimiter=",", dtype=np.float32, max_rows=25000)))
    train_tensorX = torch.from_numpy(DS[:, :-1]).to("cuda:0")
    medianANDstdArr = makeStandart(train_tensorX, numOfPeriodsPerFeature)
    train_tensorY = torch.from_numpy(DS[:, -1])
    train_dataset = TensorDataset(train_tensorX.to("cpu"), train_tensorY)
    trainer.fit(Pytorch_lightning_MNIST_my, DataLoader(train_dataset, shuffle=True, batch_size=10000, num_workers=num_of_threads))
</code></pre>
<p>Is it supposed to be like this?</p>

username: SAMOKILLER
cooked: <p>I’m sorry I didn’t immediately realize there was a separate forum for lightning.<br>
The answer is here: <a href="https://forums.pytorchlightning.ai/t/why-does-the-dataloader-run-multiple-times-and-take-up-all-the-ram/2016" class="inline-onebox" rel="noopener nofollow ugc">Why does the dataloader run multiple times and take up all the RAM? - Lightning AI</a></p>
Error in scoring a loaded model in C++

username: mmetwalli
cooked: <p>I loaded my torch script, and I was trying to score an input using it, and It is not working for some reason, and I don’t see an error message. Here is the link for the issue on GitHub for more details<br>
<a href="https://github.com/pytorch/pytorch/issues/88910" rel="noopener nofollow ugc">Issue Link</a></p>
<p>CMakeLists.txt code</p>
<pre><code class="lang-auto">cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(nn_optimization)

find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")

add_executable(nn_optimization nn_optimization.cpp)
target_link_libraries(nn_optimization "${TORCH_LIBRARIES}")
set_property(TARGET nn_optimization PROPERTY CXX_STANDARD 14)
</code></pre>
Can i fine tune a model that was trained with normal pytorch module by using pytorch lightning module?

username: Manjuphoenix
cooked: <p>I’m new to pytorch lightning &amp; I have already trained a model with normal nn module of pytorch. But i want to implement features like Model checkpoint and Early stopping. For this i thought of fine tuning the old model with pytorch lightning. Is that possible and is there any similar solutions which you have come accross??</p>
Cuda.get_device_properties(0).total_memory gives larger value than cuda.get_device_properties(0)

username: David1
cooked: <p>The line <code>torch.cuda.get_device_properties(0)</code> gives me the output<br>
<code>_CudaDeviceProperties(name='NVIDIA A100-PCIE-40GB', major=8, minor=0, total_memory=40396MB, multi_processor_count=108)</code>,<br>
but <code>torch.cuda.get_device_properties(0).total_memory / 1000000</code> gives me the output<br>
<code>42358.472704</code>.</p>
<p>Why is there a difference in ‘total memory’ here?</p>

username: ptrblck
cooked: <p>The difference would come from dividing by 1000 vs. 1024:</p>
<pre><code class="lang-python">42358.472704 * 1000000 / 1024**2
# 40396.1875
</code></pre>

username: David1
cooked: <p>Oh, I frankly didn’t know that 1 "kilo"byte is 1024 bytes. You learn something new every day. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
PyTorch build in conda environment with poetry

username: brendanbennett
cooked: <p>I’m trying to build PyTorch from source so that it uses the system CUDA and cuDNN installs. I am using PyTorch, ONNX and Tensorflow on the same docker image so it makes sense for me for PyTorch to use the same libraries while minimising image size (I could use nightly builds but this defeats the objective of minimising size).<br>
It seems to building fine into a conda env. My problems come when trying to use Poetry to install some packages that depend on torch. No matter whether I build the latest 1.13 or 1.12.1, poetry insists on ‘upgrading’ my built package to the 1.12.1 stable version. I have a feeling that there is something clever I need to do with poetry to make it recognise the built package as satisfying requirements (I’ve checked the dependencies and they only specify torch&gt;=1.4). Also, it might be to do with the fact that the built version number is <code>1.12.0a0+git664058f</code> which might not be parsed correctly by poetry.<br>
Has anyone got better experience with poetry than me that could help?</p>

username: marksaroufim
cooked: <p>You might find what you’re looking for here <a href="https://github.com/python-poetry/poetry/issues/6409" class="inline-onebox" rel="noopener nofollow ugc">Instructions for installing PyTorch · Issue #6409 · python-poetry/poetry · GitHub</a></p>
How to run PyTorch model on NodeJS Linux server?

username: RandGor
cooked: <p>Hello!<br>
I am trying to run my classifier model on my server.<br>
Can you please tell me how can I do it? Aiming the solution without spawning python processes.<br>
I will be glad to any answers.</p>
<p>BTW. I already tried to use but there are: [compilation unfixable issues/old PT ver/not works on Linux]:<br>
vvmnnnkv/libtorchjs<br>
torch-js/torch-js<br>
arition/torch-js<br>
Techainer/torch-js</p>

username: marksaroufim
cooked: <p>Yeah we don’t have a great native solution to running PyTorch on the web so your best bet is <a href="https://onnxruntime.ai/docs/tutorials/web/" class="inline-onebox" rel="noopener nofollow ugc">Deploy on web - onnxruntime</a></p>
Compare Deployment Types To Deploy A Model

username: Duc_Nguyen
cooked: <p>Hi,</p>
<p>After training a model, I want to deploy this model in production. But there are many kinds of types such as TorchScript, ONNX, and TorchServe.<br>
I read about them but it is quite difficult to understand all of them.<br>
So could you explain them easily and compare them? Which one is the best for production?</p>
<p>ps: I tested inference normally by a pytorch module and a torchscipt model in the python environment. The inference time of the pytorch module is faster. It’s quite hard to understand.</p>
<p>Help me, thanks</p>

username: Duc_Nguyen
cooked: <p>Does anybody help me? Please.</p>

username: marksaroufim
cooked: <p>PyTorch is an eager mode framework which means it runs python code line by line, this is amazing if your’e debugging your model but if you want optimal performance its less than ideal because of python overhead. Torchscript is subset of the python language that allows you to run PyTorch models without needing Python on your machine. Torchscript also acts as a good intermediate language that other optimization runtimes like ONNX can leverage to run your models faster.</p>
<p>Torchserve is a model serving tool which lets you deploy a PyTorch model regardless of whether its an eager mode pytorch model or torchscript or onnx.</p>
My python app contains "Auto-Model-For-Sequence-Classification" which requires Pytorch. Can it be deployed as a web app on Heroku or GitHub as it uses Pytorch?

username: Shaw_B
cooked: <p>A newbie here.<br>
I just made a Python app (something like a twitter sentiment analysis kindof thing) and realized it needs Pytorch for AutoModelForSequenceClassification module. Pytorch is almost 200 MB alone. Can it be deployed as a web application without any troubles?</p>

username: marksaroufim
cooked: <p>Yes that should be totally fine but FWIW we help you manage some of these complexities here <a href="https://github.com/pytorch/serve" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/serve: Serve, optimize and scale PyTorch models in production</a></p>
<p>I haven’t tried running PyTorch with Heroku but we install PyTorch all the time to run our CI in Github Actions and it works great</p>
TorchServe - example of model for time series prediction

username: DominiK_DK
cooked: <p>Hello.<br>
Does anybody could help me with any example of custom handler for model for TorchServe to be used for time-series prediction/forecasting ?<br>
Trying to create my own one, but still missing any example to learn.</p>
<p>Kind regards<br>
Dominik</p>

username: marksaroufim
cooked: <p>Time series are interesting for torchserve because the elements of a batch are related to each other. If you like we already have an example for real time video inference which should be a good starting point <a href="https://github.com/pytorch/serve/tree/master/examples/image_classifier/near_real_time_video" class="inline-onebox" rel="noopener nofollow ugc">serve/examples/image_classifier/near_real_time_video at master · pytorch/serve · GitHub</a> and if you’d like to contribute the time series example I’d be happy to merge</p>
Does torch.onnx support if control flow?

username: snsun
cooked: <p>Hi all,</p>
<p>I want to export my RNN Transducer model using torch.onnx.export. However, there is an “if” in my network forward. I have checked the documents and one way to solve the control flow problem is mixing tracing and scripting.</p>
<p><a href="https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/onnx.html#tracing-vs-scripting</a></p>
<p>I have tried the example and it works well for “for loop”, but there will be errors as following when I use “if”. So I want to know if "if " is supported.</p>
<p>My pytorch version is 1.4.0</p>
<blockquote>
<p>graph(%input : Long(2, 3),<br>
%loop : Long(),<br>
%d : Long()):<br>
%3 : Long() = onnx::Constant<a>value={3}</a><br>
%4 : Long() = onnx::Greater(%d, %3) # test2.py:8:7<br>
%5 : Long(2, 3) = onnx::If(%4) # test2.py:8:4<br>
block0():<br>
%6 : LongTensor = onnx::Add(%input, %d) # test2.py:10:12<br>
 → (%6)<br>
block1():<br>
 → (%input)<br>
return (%5)</p>
<p>Traceback (most recent call last):<br>
File “test2.py”, line 31, in <br>
ort_sess = ort.InferenceSession(‘loop.onnx’)<br>
File “/ceph/sz_ts80_new/siningsun/pytorch/espnet/tools/venv/lib/python3.7/site-packages/onnxruntime/capi/session.py”, line 158, in <strong>init</strong><br>
self._load_model(providers)<br>
File “/ceph/sz_ts80_new/siningsun/pytorch/espnet/tools/venv/lib/python3.7/site-packages/onnxruntime/capi/session.py”, line 177, in _load_model<br>
self._sess.load_model(providers)<br>
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Failed to load model with error: /onnxruntime_src/onnxruntime/core/graph/graph.cc:912 void onnxruntime::Graph::InitializeStateFromModelFileGraphProto() This is an invalid model. Graph output (input) does not exist in the graph.</p>
</blockquote>

username: hitbuyi
cooked: <p>I also want to know some explanation of this problem</p>

username: marksaroufim
cooked: <p>Yes control flow is supported but you have to export your model via scripting instead of tracing</p>
Parallel computation using cpu and gpu

username: Kins1ley
cooked: <pre><code class="lang-python">epe = self.checker(image[2].cpu())
image[0] = torch.sigmoid(image[0])
image[1] = torch.sigmoid(image[1])
image[2] = torch.sigmoid(image[2])
</code></pre>
<p>Is it possible to achieve parallel CPU and GPU compuation in pytorch?</p>
<p>I hope to calculate the epe with CPU. Epe is a metric of the image and it includes multiple for loop operations, difficult to parrallel. As for later operations, they can be calculated with gpu. The images are stored with torch.tensor. My question is whether is it possible to achieve cpu and gpu computation parallelly since the epe will not change the image[2] and the later operations.</p>

username: marksaroufim
cooked: <p>GPU operations are asynchronous so any layer or model that has a <code>.cuda()</code> will run on GPU</p>
Possible to use torch::optim with LibTorch-Lite?

username: Kapsy
cooked: <p>This may be a dumb question, but I’m in the process of converting some Python PyTorch code to run on mobile.</p>
<p>Are the following <code>torch::optim</code> classes (or equivalent functionality) supported in LibTorch-Lite?</p>
<pre><code class="lang-auto">optimizer = torch.optim.Adam(parameter_list, ...)
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda ...
</code></pre>
<p>If not, would it be possible to build libtorch (not Lite) for running on mobile so I can use these classes?</p>
<p>I’ve searched around for similar questions but could find a solid answer to the above. Any help much appreciated.</p>
Does PyTorch support if-while statements when exporting to ONNX model

username: hitbuyi
cooked: <p>my model have if-else or while statements in foward,  does PyTorch support such control statements?</p>

username: marksaroufim
cooked: <p>Yes as long as you export your model using the script based exporter <a href="https://glaringlee.github.io/onnx.html" class="inline-onebox" rel="noopener nofollow ugc">torch.onnx — PyTorch master documentation</a></p>
Padding a list of sparse tensors

username: blue123
cooked: <p>I have a list of csr sparse matrices:</p>
<pre><code>X:
[&lt;9x15466 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
 	with 115 stored elements in Compressed Sparse Row format&gt;,
 &lt;201x15466 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
 	with 4430 stored elements in Compressed Sparse Row format&gt;,
 &lt;98x15466 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
 	with 2567 stored elements in Compressed Sparse Row format&gt;,
 &lt;24x15466 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
 	with 368 stored elements in Compressed Sparse Row format&gt;,
 &lt;55x15466 sparse matrix of type '&lt;class 'numpy.float64'&gt;'
 	with 904 stored elements in Compressed Sparse Row format&gt;]
</code></pre>
<p>Which I convert to torch tensors<br>
X = sp.vstack(X).tocoo()<br>
X = sparse_coo_to_tensor(X)</p>
<pre><code>X:
tensor(indices=tensor([[    0,     0,     0,  ...,   386,   386,   386],
                       [ 1738,  1740,  3404,  ..., 15457, 15461, 15465]]),
       values=tensor([1., 1., 1.,  ..., 1., 1., 1.]),
       size=(387, 15466), nnz=8384, layout=torch.sparse_coo)

X[0]:
tensor(indices=tensor([[ 1738,  1740,  3404,  3405, 15427, 15431, 15436, 15437,
                        15445, 15448, 15453, 15457, 15464, 15465]]),
       values=tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.]),
       size=(15466,), nnz=14, layout=torch.sparse_coo)
</code></pre>
<p>I want to be able to use <code>pad_sequence</code> in my <code>collate_fn()</code>, however, <code>pad_sequence</code> doesn’t seem to work with <code>torch.sparse_coo</code> layout</p>
<pre><code>torch.nn.utils.rnn.pad_sequence(X)
</code></pre>
<blockquote>
<p>D:\bo\envs\bd\lib\site-packages\torch\nn\utils\rnn.py in<br>
pad_sequence(sequences, batch_first, padding_value)<br>
365         out_dims = (max_len, len(sequences)) + trailing_dims<br>
366<br>
 → 367     out_tensor = sequences[0].new_full(out_dims, padding_value)<br>
368     for i, tensor in enumerate(sequences):<br>
369         length = tensor.size(0)</p>
<p>RuntimeError: full(…) is not implemented for sparse layout</p>
</blockquote>
<p>Does anyone have any experience in working with padding sparse tensors and making it work for models such as RNNs, Transformers etc? Do I have to convert them to dense tensors? My motivation of using these sparse tensors in the first place is to reduce the OOM errors as I have a lot of very high dimensional feature vectors. Thanks!</p>
C python interface

username: Gongyuan_Situ
cooked: <p>Hi, i’m very interested in the implementation details of pytorch, i’ve been trying to read some source code of pytorch but i cannot seem to find a good  way, can anyone show me how pytorch integrate python with c/c++? why not use numpy for numerical computation but use c/c++ dynamic lib instead? For extending python with c/c++, which is the best choice, cffi, ctype, cython? Thanks very much!</p>

username: smth
cooked: <blockquote>
<p>why not use numpy for numerical computation but use c/c++ dynamic lib instead?</p>
</blockquote>
<p>Because numpy does not have GPU support.</p>
<blockquote>
<p>For extending python with c/c++, which is the best choice, cffi, ctype, cython?</p>
</blockquote>
<p>Internally we use our own C wrapper engine: <a href="https://github.com/pytorch/pytorch/tree/master/tools/cwrap">https://github.com/pytorch/pytorch/tree/master/tools/cwrap</a></p>
<p>For end-users, we recommend that they use CFFI <a href="https://github.com/pytorch/extension-ffi" class="inline-onebox">GitHub - pytorch/extension-ffi: Examples of C extensions for PyTorch</a></p>

username: Wesley_Neill
cooked: <p>Doesn’t CFFI not support C++?</p>
<p>I’m looking at starting a project that at the outset appears to need</p>
<ul>
<li>lua (for interacting with adobe lightroom)</li>
<li>C++ (for interacting with Adobe’s XML SDK)</li>
<li>Python (For building models in PyTorch)</li>
</ul>
<p>I think this will be my biggest attempted project ever, and I only know a little C++ and no Lua.</p>
<p>Any advice?</p>
<p>Also, why do you guys use your own wrapper engine? Do the common ones not meet your needs?</p>
How to properly negate loss function

username: Jakub_Mitura
cooked: <p>Hello I have two cross entropy loss functions A and B<br>
the result is best when A is small and B as big as possible hence I could write it as A-B<br>
However, by writing like that I do not have a lower bound - loss can get to arbitrary large negative numbers and I am training on float 16 so I am afraid that numerical instability will be the issue<br>
Another possibility is A + 1/B but I can not guarantee that B will not be 0<br>
What is the best way to deal with it?</p>

username: KFrank
cooked: <p>Hi Jakub!</p>
<aside class="quote no-group quote-modified" data-username="Jakub_Mitura" data-post="1" data-topic="165749" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jakub_mitura/48/45404_2.png" class="avatar"> Jakub_Mitura:</div>
<blockquote>
<p>I could write it as A-B<br>
…<br>
Another possibility is A + 1/B</p>
</blockquote>
</aside>
<p>Without knowing your use case nor where <code>A</code> and <code>B</code> come from, it’s hard<br>
to recommend a particular choice.</p>
<p>However, <code>torch.exp (-B)</code> would solve your immediate problem.  It becomes<br>
smaller as <code>B</code> becomes larger and is bounded below by zero.</p>
<p>If you were concerned about <code>exp (-B)</code> overflowing to <code>inf</code> as <code>B</code> becomes<br>
large and negative, you could use <code>torch.sigmoid (-B)</code>.  It is bounded<br>
above by one and approaches <code>exp (-B)</code> (and hence zero) as <code>B</code> becomes<br>
large and positive.</p>
<p>Best.</p>
<p>K. Frank</p>

username: HACS
cooked: <p>The likelihood is a number between 0 and 1, and the cross_entropy_loss is -ln(likelihood).</p>
<p>For A and B, we have likelihood_A = exp(-A) and likelihood_B = exp(-B).</p>
<p>In general, we want to maximize the likelihood (i.e. minimize the cross_entropy_loss), but from what I understood the opposite is true for likelihood_B.</p>
<p>For this case, my suggestion is to maximize likelihood_A * (1 - likelihood_B), that is, to minimize A - ln(1 - exp(-B)).</p>
<p>Any bounded, continuous, and strictly increasing function of A - ln(1 - exp(-B)) works too (e.g. tanh).</p>
Graph disconnected

username: OuisYasser
cooked: <p>hello,<br>
I have created a model in order to do <strong>binary</strong> classification, However the model I created returns a vector and based on this vector I returns either torch.FloatTensor([0]) or torch.FloatTensor([1])<br>
so forward function is as follows:</p>
<pre><code class="lang-auto">def forward(self, x):
      x = self.model(x)  #output is [n, 1]
      x = self.function_check(x)  #output is a new vector [m, 1]
      if x.sum() == 10:
                return torch.FloatTensor([0]).requires_grad_()
      else:
                return torch.FloatTensor([1]).requires_grad_()
</code></pre>
<p>so I am afraid that using this technique, my graph of output is not connected to my model and therefore, the gradients will not backpropagated through the model. am I right ?</p>
<p>Edit: I displayed the graph using torchviz, and I was right my graph is disconnected when I am returning new labels, so my question now is how can I attach my output to the graph</p>

username: HACS
cooked: <p>My suggestion is</p>
<p>x = torch.ones(9, dtype=torch.float32, requires_grad=True).view(-1, 1)</p>
<p>y = x.sum()</p>
<p>z = torch.sign(torch.abs(y - 10))</p>
<p>z</p>
<p>Output (I removed &lt; before SignBackward0): tensor(1., grad_fn=SignBackward0&gt;)</p>
Data size in LSTM

username: astra1234567
cooked: <p>Hi ! I am working with LSTM network and I am thinking about way of send data to network. So, I have tensors with shape : [ 64, 1, 8000 ] (64 is size of batch). Is one single tensor too long (8000) for one-time sending ? Should I group it or maybe something other ?  Is there automatic process in nn.LSTM for group data ?</p>
How to measure execution time in PyTorch?

username: suraj.pt
cooked: <h3>
<a name="tldr-1" class="anchor" href="#tldr-1"></a>tl;dr</h3>
<p>The recommended profiling methods are:</p>
<h5>
<a name="torchprofiler-2" class="anchor" href="#torchprofiler-2"></a>torch.profiler:</h5>
<ul>
<li><a href="https://pytorch.org/docs/master/profiler.html">API Docs</a></li>
<li><a href="https://pytorch.org/tutorials/beginner/profiler.html">Profiler Tutorial</a></li>
<li><a href="https://pytorch.org/tutorials/recipes/recipes/profiler_recipe.html">Profiler Recipe</a></li>
</ul>
<h5>
<a name="torchutilsbenchmark-3" class="anchor" href="#torchutilsbenchmark-3"></a>torch.utils.benchmark:</h5>
<ul>
<li><a href="https://pytorch.org/docs/master/benchmark_utils.html">API docs</a></li>
<li><a href="https://pytorch.org/tutorials/recipes/recipes/benchmark.html">Benchmark Recipe</a></li>
</ul>
<h3>
<a name="cpu-only-benchmarking-4" class="anchor" href="#cpu-only-benchmarking-4"></a>CPU-only benchmarking</h3>
<p>CPU operations are synchronous; you can use any Python runtime profiling method like <code>time.time()</code>.</p>
<h3>
<a name="cuda-benchmarking-5" class="anchor" href="#cuda-benchmarking-5"></a>CUDA benchmarking</h3>
<p>Using <code>time.time()</code> alone won’t be accurate here; it will report the amount of time used to launch the kernels, but not the actual GPU execution time of the kernel. Passing <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.synchronize"><code>torch.cuda.synchronize()</code></a> waits for all tasks in the GPU to complete, thereby providing an accurate measure of time taken to execute.</p>
<pre><code class="lang-auto">train() # run all operations once for cuda warm-up
torch.cuda.synchronize() # wait for warm-up to finish

times = []
for e in range(epochs):
    torch.cuda.synchronize()
    start_epoch = time.time()
    train()
    torch.cuda.synchronize()
    end_epoch = time.time()
    elapsed = end_epoch - start_epoch
    times.append(elapsed)

avg_time = sum(times)/epochs
</code></pre>
Trouble pruning model weights on model with many submodules

username: bigtuna
cooked: <p>I am trying to prune a pretrained huggingface model from the transformers library, but the code below is giving me an error.</p>
<pre><code class="lang-auto">checkpoint = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSequenceClassification.from_pretrained(checkpoint)

for module in model.named_modules():
    prune.global_unstructured(module, pruning_method = prune.L1Unstructured, amount = 0.9)
</code></pre>
<p>gives</p>
<pre><code class="lang-auto">ValueError: not enough values to unpack (expected 2, got 0)
</code></pre>
<p>Is there a way to prune all the model weights, accounting for the fact that most of the “layers” in the model are contained in larger modules with many other layers?</p>
Saving torch.Distributions with state_dict()

username: zrene
cooked: <p>I am using the torch.Distribution package inside of my Module.<br>
However, since the distribution classes do not inherit from torch.Module, they won’t be saved when calling my_module.state_dict().</p>
<p>Is there an easy way to save the distributions?</p>
<pre><code class="lang-auto">import torch

class GmmNet(torch.nn.Module):
    def __init__(self):
        super(GmmNet, self).__init__()
        # Example, random parameters
        self.gmm = D.MixtureSameFamily(D.Categorical(torch.nn.parameter.Parameter(torch.ones(5,))), D.Normal(torch.randn(5,),(torch.rand(5,))))

    def forward(self, x):
        return self.gmm.log_prob(x)
        
print(GmmNet().state_dict())
# --&gt; OrderedDict()
</code></pre>

username: googlebot
cooked: <p>bigger issue is that Distribution objects are immutable, so you can’t push nn.Parameters inside them in <code>__init__</code>. Keep parameters in modules, create transient Distribution objects in forward, then saving &amp; training will work correctly.</p>

username: zrene
cooked: <p>Thanks!</p>
<p>For anyone else, this is the working code:</p>
<pre><code class="lang-auto">import torch

class GmmNet(torch.nn.Module):
    def __init__(self):
        super(GmmNet, self).__init__()
        self.cat = torch.nn.parameter.Parameter(torch.ones(5,))
        self.cov = torch.nn.parameter.Parameter(torch.ones(5,))

    def forward(self, x):
        gmm = D.MixtureSameFamily(D.Categorical(self.cat), D.Normal(self.cov, self.cov))
        return gmm.log_prob(x)
        
print(GmmNet().state_dict())
# --&gt; OrderedDict([('cat', tensor([1., 1., 1., 1., 1.])), ('cov', tensor([1., 1., 1., 1., 1.]))])
</code></pre>

username: Mauricio_Maroto
cooked: <p>Hi <a class="mention" href="/u/zrene">@zrene</a> Will this approach work for saving a a pyro transformed distribution for deployment?</p>
Torch.nn.functional.pad return tensor

username: nitaifingerhut
cooked: <p>Hi all, does <code>torch.nn.functional.pad</code> return a view of the tensor or a new tensor?</p>

username: khushi-411
cooked: <p>Hi,<br>
To be more precise, <code>view</code> (<a href="https://pytorch.org/docs/stable/generated/torch.Tensor.view.html" rel="noopener nofollow ugc">Doccumentation</a>) itself returns a new tensor, but it can only be applied for contiguous tensors. So I’m assuming you wanted to know the memory format of the output tensor w.r.t the input tensor. Is that the same you wanted to know?</p>
<p>You can check this using the <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.is_contiguous.html" rel="noopener nofollow ugc">is_contiguous()</a> operator in PyTorch. <code>torch.nn.functional.pad</code> returns a new tensor with the contiguous memory format. Please see:</p>
<pre><code class="lang-python">In [1]: import torch

In [2]: from torch.nn import functional as F

In [3]: t4d = torch.empty(3, 3, 4, 2)

In [4]: p1d = (1, 1)

In [5]: out = F.pad(t4d, p1d, "constant", 0)

In [6]: out.is_contiguous()
Out[6]: True
</code></pre>
<p>Feel free to ping me if it’s not clear to you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thanks!</p>
PyTorch with vulkan backend

username: vjayd
cooked: <p><a class="mention" href="/u/ivankobzarev">@IvanKobzarev</a> do we have a libtorch for vulkan backend?<br>
Also, I tried running .ptl file generated with backend=‘vulkan’ on a Android phone with c++ interface. I see that the tensor.device().is_vulkan() returns True (tensors are getting copied to vulkan) but when I run the module.forward() it throws an error Throws Could not run ‘prepacked::conv2d_clamp_prepack’ with arguments from the ‘vulkan’ backend. Can you help me here?</p>
Add softmax layer to vision transformer model

username: vogelstrauss
cooked: <p>I am trying to add a softmax layer to a <code>vit-b</code> model with 10 outputs.<br>
While it works calling <code>torch.softmax</code> directly on the model output, I do not get softmax output when adding the softmax as a fully connected layer:</p>
<pre><code class="lang-auto">import torchvision
import torch

torch.manual_seed(1)
image = torch.rand(1,3,224,224)
model = torchvision.models.vit_b_16(weights="DEFAULT")
model.heads.head = torch.nn.Linear(in_features=768, out_features=10, bias=True)
model.fc = torch.nn.Softmax(10)
model(image)
</code></pre>
<p>This does not seem to apply softmax, the output is</p>
<pre><code class="lang-auto">tensor([[ 0.0204, -0.0550, -0.7651, -0.2418,  0.0715,  0.1525, -0.1907, -0.1199,
          0.0427, -0.2585]], grad_fn=&lt;AddmmBackward0&gt;)
</code></pre>
<p>What am I doing wrong?</p>

username: ptrblck
cooked: <p><code>vit_b_16</code> doesn’t use a <code>.fc</code> module so you are creating a new attribute in:</p>
<pre><code class="lang-python">model.fc = torch.nn.Softmax(10)
</code></pre>
<p>which is never used.<br>
You could replace the <code>model.heads.head</code> with an <code>nn.Sequential</code> container containing the new <code>nn.Linear</code> as well as the <code>nn.Softmax</code> layer.</p>

username: vogelstrauss
cooked: <p>Indeed, this worked.</p>
<pre><code class="lang-auto">model.heads.head = torch.nn.Sequential(torch.nn.Linear(in_features=768, out_features=10, bias=True), torch.nn.Softmax(dim=1))
</code></pre>
<p>Thank you very much for your help!</p>
How to train with fp16 in torch.cuda.amp?

username: doyi_kim
cooked: <p>How can I train a model with only fp16?<br>
the same operation with apex opt_level=“03” not mixed precision</p>

username: ptrblck
cooked: <p>The deprecated <code>apex.amp</code> <code>opt_level="O3"</code> was using “pure” FP16, so you can just call <code>.half()</code> on your model and input data in your training script.</p>

username: doyi_kim
cooked: <p>if so, should I remove the amp.auto-cast() and gradscale?<br>
also is there any way to converge model? the model only predicts zero or one in binary classification…</p>

username: ptrblck
cooked: <p>Yes, you won’t need to use <code>autocast</code>, since you are explicitly skipping the mixed-precision training util. Note that pure FP16 training is generally not stable for training.</p>
Libtorch and gRPC conflicts. How to install?

username: woong97
cooked: <p>I faced similar issue in <a href="https://discuss.pytorch.org/t/conflict-between-libtorch-and-grpc/54627" class="inline-onebox">Conflict between libtorch and grpc</a><br>
If I link only one among libtorch and grpc,  it works. However, If I tried to link both at the same time, build failed.</p>
<pre><code class="lang-auto">/usr/bin/ld: CMakeFiles/server.dir/server.cc.o: in function `RunServer()':
server.cc:(.text+0x139): undefined reference to `grpc::ServerBuilder::AddListeningPort(std::string const&amp;, std::shared_ptr&lt;grpc::ServerCredentials&gt;, int*)'
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/server.dir/build.make:150: server] Error 1
</code></pre>
<pre><code class="lang-auto">Environment: Ubuntu 20.04
libtorch: 1.11.0+cpu 
gRPC: 1.50.0 ( And I tried 1.40, 1.38) 
</code></pre>
<p>In previous issues which I linked, there was comment, *“Currently the easiest way is to compile libtorch with the protobuf library that grpc uses, or compile grpc with the protobuf library that libtorch uses”.</p>
<p>So I tried to install grpc with the protobuf library that libtorch uses.  The process as follows,</p>
<pre><code class="lang-auto"># Install torch 1.11.0
wget https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.11.0%2Bcpu.zip
unzip libtorch-shared-with-deps-1.11.0+cpu.zip &amp;&amp; rm libtorch-shared-with-deps-1.11.0+cpu.zip

# Install gRPC
git clone -b v1.50.0 https://github.com/grpc/grpc
cd grpc
git submodule update --init
cd third_party
rm -r protobuf   # remove protobuf and clone protobuf version 3.13.0.1 that libtorch uses
git clone --recurse-submodules -b v3.13.0.1 --depth 1 --shallow-submodules https://github.com/protocolbuffers/protobuf.git

mkdir -p cmake/build &amp;&amp; \
        cd cmake/build &amp;&amp; \
        cmake -DgRPC_INSTALL=ON \
              -DgRPC_BUILD_TESTS=OFF \
              -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE \
              -DCMAKE_INSTALL_PREFIX=/usr/local/grpc \
              ../.. &amp;&amp; \
        make &amp;&amp; \
        make install &amp;&amp; \
        ldconfig
</code></pre>
<p>How can I resolve this problem??? Is it correct way to install protobuf that libtorch uses?</p>
Get_graph_node_names function call on FASTERRCNN_RESNET50_FPN_V2

username: sj_7.0
cooked: <p>I am trying to create a feature extractor using the FASTERRCNN_RESNET50_FPN_V2 model. Specifically I want to get the <strong>box_head</strong> outputs for each input image. I understand that the FX has something called <strong>leafs</strong> to enable this functionality but I haven’t been successful to get it done. Any ideas as to how this can be implemented?</p>
<p>Thanks.</p>
Undefined reference to `c10::Scalar at::Tensor::item<c10::Scalar>() const'

username: Shobhit_Narayanan
cooked: <p>Hi,</p>
<p>So I am trying to obtain a value from a tensor and convert it to c10::Scalar so that I can pass the value to at::arange(). I have tried many combinations of this. How can I do it?</p>
<p><code>auto s = e[0].item&lt;torch::Scalar&gt;();</code></p>
How to train a model with fixed precision in some layer?

username: beann
cooked: <p>I want to train a model and set the first layers to fixed precision (e.g., 8-bit).  Is this possible?</p>
<p>A concrete example, suppose I want to train an Alexnet, but set the initial 4 layers to 8-bit and the others as 32-bit.</p>
<p>I notice there is <a href="https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html" rel="noopener nofollow ugc">Dynamic quantilization</a>. But it only supports 32-bit input and does not support training in this process. Similarly, QAT does not support different precision for different layers.</p>
Is torch.distributed.all_reduce implemented with Ring-AllReduce or Tree-based AllReduce, or others?

username: steamgjk
cooked: <p>I am using Gloo as the backend for distributed machine learning. I am curious about the implementation of torch.distributed.all_reduce in detail. Currently the official documentation does not talk about it. I wonder whether it is a ring-based all-reduce or tree-based all-reduce?</p>

username: zizhao.mo
cooked: <p>Hi, I have the same problem. Could anyone answer this problem?</p>
Can I customize a new data type? For example, I want to use 4 bits to store integers in pytorch, but there is no int4 in torch. How can I write an int4 data type?

username: fqyld
cooked: <p>I want to do some distributed training, in which some integers need to be transmitted, and only a few bits are needed to represent them (such as labels, quantized integers in communication compression). However, there are only 8-bit integers in torch. I can use bytes to encode integers to reduce communication overhead, but this is too time-consuming. Therefore, if I can set the data type of the specified bit myself, I can solve my problem. How do I customize data types?</p>
PyTorch's `torch.cuda.max_memory_allocated()` showing different results from `nvidia-smi`?

username: seankala
cooked: <p>I’m currently making my own custom GPU report and am using <code>torch.cuda.max_memory_allocated(device_id)</code> to get the maximum memory that each GPU uses. However, I’ve noticed that this number is different from when I run <code>nvidia-smi</code> during the process.</p>
<p>According to the <a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_allocated.html#torch-cuda-max-memory-allocated" rel="noopener nofollow ugc">documentation for <code>torch.cuda.max_memory_allocated</code></a>, the output integer is in the form of bytes. And from what I’ve searched online to convert the number of bytes to the number of gigabytes, you should divide it by <code>1024 ** 3</code>. I’m currently doing <code>round(max_mem / (1024 ** 3), 2)</code></p>
<p>Am I doing the calculation wrong, or am I misunderstanding how <code>torch.cuda.max_memory_allocated</code> works entirely? The memory allocated I’ve observed from one GPU during the entire process was aroudn 32GB, but <code>torch.cuda.max_memory_allocated(0) / (1024 ** 3)</code> returns aroudn 13.5GB.</p>

username: eqy
cooked: <p>It’s not expected for these two values to match for several reasons. Because the current CUDA memory allocator is a caching allocator, more memory than is currently occupied by tensors (reported by <code>torch.cuda.max_memory_allocated</code>) will be “reserved” to speed up future allocations/reclaim unused memory from garbage collected tensors. The reserved memory <a href="https://pytorch.org/docs/stable/generated/torch.cuda.max_memory_reserved.html#torch.cuda.max_memory_reserved" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.max_memory_reserved — PyTorch 1.13 documentation</a> will be closer to what is reported by <code>nvidia-smi</code>, but keep in mind that there will be some additional overhead depending on what operations/libraries are being used as e.g., (cuDNN, cuBLAS, etc. kernels will use memory). The latter can be substantial (on the order of hundreds of MiB) in many cases.</p>
Do I need to implement gradient for this?

username: IZugec
cooked: <p>Imagine a situation in which my neural network takes coordinates of the planets at some time <em>t</em>, and outputs coordinates at a time <em>t</em>+Δ<em>t</em>. Due to the nature of my simulation which I am using to train this network, it’s much more convenient to compare distances of one fixed planet to all others than directly all coordinates which is why in a loss function I would like to have a term like</p>
<p><strong>L</strong> =  sum_over_batch [ (Distances_by_model - Distances_by_ground_truth)**2 ]</p>
<p>Just to make clear, distance is a function of <strong>coordinates of planets</strong> which maps an array of coordinates of shape (N_planets,3) to a vector (N_planets,1).</p>
<p>Finally, my question is: since my NN outputs coordinates, and not distances, but I have distances in a loss functions, do I have to implement autograd distance function? If not, could you name an example in which I would have to implement it?</p>
<p>Thanks in advance,<br>
confused physics student</p>

username: eqy
cooked: <p>If you can express your loss function as a sequence of differentiable operations on the ground truth and the output of the model that exist in PyTorch, then you would not need to implement your own autograd distance function. As a toy example, if your loss function is something like (((model_x1 - modelx2)**2 + (model_y1 - modely2)**2) - (ground_truth_val1))**2 then you would not need to write your own autograd function as the basic operations of addition, subtraction, and squaring are all differentiable and have their own autograd implementations that can be chained together.</p>
<p>However, if in the future you wanted to fuse your loss function into a single kernel for higher performance (speed/efficiency) then you might want to look into writing your own autograd function + custom kernel. Otherwise it wouldn’t be necessary.</p>
No CMAKE_CXX_COMPILER could be found

username: mmetwalli
cooked: <p>[variant] Loaded new set of variants</p>
<p>[kit] Successfully loaded 9 kits from C:\Users\mmetwalli\AppData\Local\CMakeTools\cmake-tools-kits.json</p>
<p>[visual-studio] Parse '' failed</p>
<p>[proc] Executing command: “D:\Program Files\CMake\bin\cmake.EXE” -Sd:/OneDrive/Documents/Projects/pytorch-cpp -Bd:/OneDrive/Documents/Projects/pytorch-cpp/build -G “Visual Studio 17 2022”</p>
<p>[proc] The command: “D:\Program Files\CMake\bin\cmake.EXE” -Sd:/OneDrive/Documents/Projects/pytorch-cpp -Bd:/OneDrive/Documents/Projects/pytorch-cpp/build -G “Visual Studio 17 2022” exited with code: 1 and signal: null</p>
<p>[main] Configuring folder: pytorch-cpp</p>
<p>[proc] Executing command: “D:\Program Files\CMake\bin\cmake.EXE” --no-warn-unused-cli -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -Sd:/OneDrive/Documents/Projects/pytorch-cpp -Bd:/OneDrive/Documents/Projects/pytorch-cpp/build -G “Visual Studio 17 2022”</p>
<p>[cmake] Not searching for unused variables given on the command line.</p>
<p>[cmake] – Selecting Windows SDK version to target Windows 10.0.19044.</p>
<p>[cmake] – The C compiler identification is unknown</p>
<p>[cmake] – The CXX compiler identification is unknown</p>
<p>[cmake] CMake Error at CMakeLists.txt:2 (project):</p>
<p>[cmake] No CMAKE_C_COMPILER could be found.</p>
<p>[cmake]</p>
<p>[cmake]</p>
<p>[cmake]</p>
<p>[cmake] CMake Error at CMakeLists.txt:2 (project):</p>
<p>[cmake] No CMAKE_CXX_COMPILER could be found.</p>
<p>[cmake]</p>
<p>[cmake]</p>
<p>[cmake]</p>
<p>[cmake] – Configuring incomplete, errors occurred!</p>
<p>[cmake] See also “D:/OneDrive/Documents/Projects/pytorch-cpp/build/CMakeFiles/CMakeOutput.log”.</p>
<p>[cmake] See also “D:/OneDrive/Documents/Projects/pytorch-cpp/build/CMakeFiles/CMakeError.log”.</p>
<p>[proc] The command: “D:\Program Files\CMake\bin\cmake.EXE” --no-warn-unused-cli -DCMAKE_EXPORT_COMPILE_COMMANDS:BOOL=TRUE -Sd:/OneDrive/Documents/Projects/pytorch-cpp -Bd:/OneDrive/Documents/Projects/pytorch-cpp/build -G “Visual Studio 17 2022” exited with code: 1 and signal: null</p>
<p>[visual-studio] Parse '' failed</p>

username: mmetwalli
cooked: <p>I solved it by installing Windows SDK</p>