akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torchserve with websocket

username: Dennis_Hugle
cooked: <p>Dear Community,</p>
<p>I’m trying to set up a image-segmentation proof-of-concept to deploy my neural network to the cloud using torchserve.</p>
<p>What is the issue?</p>
<p>When connecting with a &gt;requests&lt; - based python client to the torchserve server, it appears that the only way to get a response is via a REST-API. That returns a json that contains the data as a list.</p>
<p>This is my client code for a put request on a running torchserve-server:</p>
<pre><code class="lang-auto">headers = {'Content-type': 'image/png', 'Slug': "your_file.png"}
r = requests.put(url, data=open(path, 'rb'), headers=headers)
print(r.json())

&gt;&gt;&gt;[ [ 0.0,  -28.528526306152344,  ...long_list] ]
</code></pre>
<p>The client receives the segmented image as a json. Both the http and the json serialization cause overhead. Is it possible to send the data bidirectional using e.g. a websocket connection?</p>
<p>Cheers</p>

username: marksaroufim
cooked: <p>Hi Dennis not as of now, but could you elaborate a bit more on the benefits of a web socket approach? Generally I’d expect the bottleneck to be the model inference and not the json and http serialization for a serving framework</p>
Serving pytorch trained model with Java server (Spring boot)

username: ya_luyanta
cooked: <p>Is the pytorch provided Java library enough for serving (HTTP) a model in production?</p>
<p>Here are libraries used:</p>
<pre><code class="lang-auto">    implementation 'org.pytorch:pytorch_java_only:1.9.0'
    implementation group: 'com.facebook.fbjni', name: 'fbjni-java-only', version: '0.2.2'
    runtimeOnly group: 'com.facebook.soloader', name: 'nativeloader', version: '0.10.1'

    implementation group: 'org.springframework.boot', name: 'spring-boot', version: '2.5.6'
</code></pre>
<p>Can people share resources and suggestions for Java backed model serving in production?<br>
I see there exists TorchServe written in Java but I don’t have control on Infrastructure to install torchserve.</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/ya_luyanta">@ya_luyanta</a> could you elaborate a bit more? You can pip install torchserve on any machine and serve models to it. I’d suspect you’ll need to install some library on a prod machine to serve models.</p>
Using TorchServe to save images (Segmentation/Super-resolution)

username: anustup
cooked: <p>Hi everyone,</p>
<p>I have seen examples of TorchServe being used for tasks such as image classification, object detection where the output are the classes or the bounding box of the objects respectively. These predictions are typically printed on the terminal. Likewise, for the Image_Segmenter case, currently the output is an array with labels for each pixel.</p>
<p>Instead, is it possible to use TorchServe to save images? For the segmentation task, this can be an image with a label for each pixel. For a super-resolution task, this could be the up-sampled image.</p>
<p>I tried modifying the handler to save the input image (For the MNIST example) but I don’t see the output file getting created. Can I get any pointers regarding this issue ?</p>
<p>Thanks</p>

username: marksaroufim
cooked: <p>Torchserve handlers are pretty general in what they can accomplish so just make sure that either in the inference or postprocess function that you’re saving the image to disk</p>
<p>If you can share your model handler it may make things easier to repro</p>
Custom transform on minibatch during training loop

username: squish
cooked: <p>I wish to make a custom transform to apply to my images… DURING the training loop<br>
I think I understand how to make transforms that apply during dataloader creation, but not during the loop.</p>
<p>Is there any guide, or method, to making a custom transform class object that can be applied during the training loop to minibatches for unsupervised learning (no labels)?</p>
<p>If you’re wondering why I can’t apply the transform during the dataloader step, it’s because I am training a generator, and need to apply the transform to the freshly-generated minibatches on each loop in order to calculate the loss function.</p>
<p>I am fairly new to pytorch, so thank you so much for any help!</p>

username: ptrblck
cooked: <p>Yes, <code>torchvision.transforms</code> can be applied on tensors in newer releases, so you could apply them inside the training loop on the data.</p>
Torchserve - how to log custom metrics?

username: vishalagarwal
cooked: <p>I was going through the TorchServe <a href="https://github.com/pytorch/serve/blob/master/docs/metrics.md" rel="nofollow noopener">documentation</a> for logging custom metrics but I’m not sure I fully understand it, especially the use of <code>Dimension</code> class. What exactly is its purpose?</p>
<p>Can someone please explain with a short toy example? Let’s say I’m serving a classification example and I want to log 2 things -</p>
<ol>
<li>Probability of the predicted class</li>
<li>Probability of the top 5 predicted classes.</li>
</ol>
<p>How can I do it? Thanks a lot!</p>

username: marksaroufim
cooked: <p>The <code>Dimension</code> class is just used to group certain metrics so you can easily filter in a dashboard tool like Prometheus. In the case of metrics we provide there’s the host or system level metrics which have things like CPU utilization.</p>
<p>For model level metrics this is what you need <a href="https://github.com/pytorch/serve/blob/master/docs/metrics.md#log-custom-metrics" rel="noopener nofollow ugc">https://github.com/pytorch/serve/blob/master/docs/metrics.md#log-custom-metrics</a></p>
Easier method for model inference and deployment

username: kleingeo
cooked: <p>I have been looking into ways to deploy and use models after they are trained. I originally started machine/deep learning using Keras/Tensorflow, and one of the options with them is the ability to save the model architecture as a <code>.json</code> file and loading the weights into that. Is there any plans on doing something like with with PyTorch? For a lot of use cases, it isn’t ideal to use the same development project to rebuild the model architecture to load the weights into in the deployment use-case.</p>

username: RicCu
cooked: <p>Have you tried using Torchscript or exporting the model to onnx? They are self contained with weights and model structure.</p>

username: kleingeo
cooked: <p>I haven’t looked too much into Onnx, but that is on my radar. Not sure what Torchscript is, I should look into that.</p>

username: marksaroufim
cooked: <p>You can take a look at <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a> to deploy models otherwise you can use an intermediate export representation of choice like torchscript or ONNX</p>
Multithreaded inference

username: MRK
cooked: <p>I have a deployment scenario where it is required that multiple python threads should be using the same model for inference.<br>
How should I modify/structure the inference code so that multiple threads can use the same mode?</p>
<p>What I am observing is that some of the threads are failing with the following error<br>
<strong>The expanded size of the tensor (100) must match the existing size (3) at non-singleton dimension 1.</strong><br>
This error does not happen when the code is run in a sequential manner.</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/mrk">@MRK</a> could you provide a bit more detail. Are you using an serving framework or id you create your own? Any code snippets you can share? Are you following instructions from this tutorial <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html" class="inline-onebox" rel="noopener nofollow ugc">CPU threading and TorchScript inference — PyTorch 1.10 documentation</a></p>
Can Pytorch serving in c++ like tfserving accept proto?

username: jun_yu
cooked: <p>TFserving use prediction_service.proto for gRPC request/response ,and I know  "LOADING A PYTORCH MODEL IN C++ " from this <a href="https://pytorch.org/tutorials/advanced/cpp_export.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_export.html</a> ,like this:</p>
<p>// Create a vector of inputs.<br>
std::vector<a>torch::jit::IValue</a> inputs;<br>
inputs.push_back(torch::ones({1, 3, 224, 224}));</p>
<p>// Execute the model and turn its output into a tensor.<br>
at::Tensor output = module-&gt;forward(inputs).toTensor();</p>
<p>std::cout &lt;&lt; output.slice(/<em>dim=</em>/1, /<em>start=</em>/0, /<em>end=</em>/5) &lt;&lt; ‘\n’;</p>
<p>and create a vector of  <code>torch::jit::IValue</code>  (a type-erased value type  <code>script::Module</code>  methods accept and return) and add a single input.<br>
<strong>Is it possible torch::jit::IValue can like PredictRequest  or PredictResponse that can be SerializeToString ParseFromString</strong> ,in that way ,so I can deploy it more conveniently</p>
<p><strong>Can libtorch C++ provide PB protocol (or any others protocol  can be Serialize and Deserialize) as  module-&gt;forward input and output parameters,or torch::jit::IValue provide Serialize and Deserialize method?</strong></p>

username: ptrblck
cooked: <p>Please don’t try to tag certain people, as this might discourage others to provide an answer.</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/jun_yu">@jun_yu</a> could you elaborate a bit more on what you’re trying to do? It seems like you’re trying to deploy a PyTorch model using TFServing by serializing jitted values? I don’t have much experience with this workflow but perhaps you’ll find an answer in the torchserve gRPC docs <a href="https://github.com/pytorch/serve/blob/master/docs/grpc_api.md" rel="noopener nofollow ugc">https://github.com/pytorch/serve/blob/master/docs/grpc_api.md</a></p>
Deploy pytorch model on spark

username: 0bff83efac608c536648
cooked: <p>I have trained a model on GPU with PyTorch on python. Now I want to deploy the model on spark environment for production, I wonder how to deploy the model on Spark.</p>

username: 0bff83efac608c536648
cooked: <ol>
<li>Maybe I should write a java/scala code to load the parameters and make the prediction</li>
<li>Maybe rewrite the code by keras and  load the model by Deeplearning4j</li>
<li>I wonder if there is a more easy way?</li>
</ol>

username: marksaroufim
cooked: <p>You have 2 options, either you run Databricks runtime for ML <a href="https://docs.databricks.com/applications/machine-learning/train-model/pytorch.html#" class="inline-onebox" rel="noopener nofollow ugc">PyTorch | Databricks on AWS</a></p>
<p>Or you can deploy a Python model as a service and use that in your spark codebase <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>
Problem running inference of models in Docker on GPU

username: tius
cooked: <p>Hello,</p>
<p>I am new to Pytorch, and I have recently been playing around with TorchServe to try to serve several convolutional models. TorchServe is run in a Docker container with a proper tag (‘gpu’ - the host system has a single RTX 3080). I have installed proper drivers and NVIDIA Container Toolkit. Also, I have built a local image with cu111 as described <a href="https://github.com/pytorch/serve/tree/master/docker" rel="noopener nofollow ugc">here</a>.</p>
<p>I wanted the models to run predictions on the GPU. As far as I understood, the only obligatory condition for it is that both the model and the input tensor are loaded to the memory of the GPU. I have verified that the tensor and the model are loaded to the memory of GPU using <code>x.is_cuda</code> (for input tensor) and <code>next(self.model.parameters()).is_cuda</code> (for the model). Nvidia-smi tool also showed that the memory of the gpu was occupied by the torchserve process.</p>
<p>However, when querying the TorchServe for inference, the prediction was done entirely on CPU (as the CPU load was close to 100%, and the gpu was idle at 0% to 2%).</p>
<p>Could anyone please give the clue what I could be missing, and what options to try to verify I am doing everything correctly to run predictions on GPU?</p>
<p>Thank you for your time, would appreciate any clues!</p>
<p>P.S. Logs of TorchServe showed the GPU was seen and recognized, no warnings related to incompatible CUDA version and etc.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="tius" data-post="1" data-topic="142953">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tius/48/31797_2.png" class="avatar"> tius:</div>
<blockquote>
<p>However, when querying the TorchServe for inference, the prediction was done entirely on CPU (as the CPU load was close to 100%, and the gpu was idle at 0% to 2%).</p>
</blockquote>
</aside>
<p>This could point towards a CPU bottleneck, e.g. in the data loading.<br>
Try to profile the code to see which operations are the actual bottleneck and to verify that the GPU is indeed used.</p>

username: tius
cooked: <p>Thanks for the hint! Had the same suspicion, will investigate…</p>

username: marksaroufim
cooked: <p>Hi Timur, were you able to resolve this issue? A repro with your mar file on some shared drive would make it easy for me to debug. As far CPU bottlenecks go are you working with large inputs or large batch sizes?</p>
Trouble understanding multiprocessing in python and PyTorch (Connection Refused)

username: Caden_Miller
cooked: <p>I new to both python and pyTorch and am trying to understand how multiprocessing works. The following code:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp

def test(q):
  t = torch.normal(mean=0.0, std=1.0, size=(2, 3))
  q.put(t)

if __name__ == "__main__":
  mp.set_start_method("spawn", force=True)

  q = mp.SimpleQueue()
  processes = []
  for _ in range(4):
    p = mp.Process(target=test, args=(q,))
    p.start()
    processes.append(p)
  for p in processes:
    p.join()
  while q.empty() == False:
    print(q.get())
</code></pre>
<p>produces the following error:<br>
Traceback (most recent call last):<br>
File “/home/cadenmiller/Documents/coding/pyTorch/mp.py”, line 20, in <br>
print(q.get())<br>
File “/usr/lib/python3.9/multiprocessing/queues.py”, line 368, in get<br>
return _ForkingPickler.loads(res)<br>
File “/home/cadenmiller/.local/lib/python3.9/site-packages/torch/multiprocessing/reductions.py”, line 289, in rebuild_storage_fd<br>
fd = df.detach()<br>
File “/usr/lib/python3.9/multiprocessing/resource_sharer.py”, line 57, in detach<br>
with _resource_sharer.get_connection(self._id) as conn:<br>
File “/usr/lib/python3.9/multiprocessing/resource_sharer.py”, line 86, in get_connection<br>
c = Client(address, authkey=process.current_process().authkey)<br>
File “/usr/lib/python3.9/multiprocessing/connection.py”, line 507, in Client<br>
c = SocketClient(address)<br>
File “/usr/lib/python3.9/multiprocessing/connection.py”, line 635, in SocketClient<br>
s.connect(address)<br>
ConnectionRefusedError: [Errno 111] Connection refused</p>
<p>I saw somewhere online that this is because the subprocess must be running until a get the tensor from the queue, but I have no idea how to do that. Any advice or suggestions would be great.</p>
Indexed copy along dimension

username: umbriel
cooked: <p>I want to copy tensor elements to another tensor at positions given by indices along given dimension.<br>
Example:<br>
A = [[3, 2, 1], [2, 4, 5], [6, 3, 2]]<br>
B = [X, Y, Z]<br>
Index = [0, 2, 1]<br>
Dim = 1</p>
<p>Result should be: [[X, 2, 1], [2, 4, Y], [6, Z, 2]]<br>
What’s the best way to achieve this?</p>
<p>Thanks!</p>

username: KFrank
cooked: <p>Hi Umbriel!</p>
<aside class="quote no-group" data-username="umbriel" data-post="1" data-topic="142827" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/u/9de053/48.png" class="avatar"> umbriel:</div>
<blockquote>
<p>Result should be: [[X, 2, 1], [2, 4, Y], [6, Z, 2]]</p>
</blockquote>
</aside>
<p>You could use indexing with assignment:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; A = torch.tensor ([[3, 2, 1], [2, 4, 5], [6, 3, 2]])
&gt;&gt;&gt; B = torch.tensor ([100, 200, 300])
&gt;&gt;&gt; Index = torch.tensor ([0, 2, 1])
&gt;&gt;&gt; A[torch.arange (3), Index] = B
&gt;&gt;&gt; A
tensor([[100,   2,   1],
        [  2,   4, 200],
        [  6, 300,   2]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: umbriel
cooked: <p>Thank you KFrank, it works!</p>
$$ RPi developer wanted - remote sensing, computer vision, machine learning

username: Nikita_Mezh
cooked: <p>Looking for a developer to review &amp; suggest RPi-based plant health sensor configurations, assemble and calibrate prototype, develop and update desktop app for morphological and physiological plant traits analysis and crop management</p>
<p>Sensor configurations range from Pi Zero-based multispectral camera to advanced ROS-enabled laser scanner with air quality sensor board and positioning system for scanning automation</p>
<p>Multispectral camera configurations use photogrammetry method to generate plant 3d model. Laser scanner and multispectral camera working together enable more precise spatial data collection, high-throughput scanning and sensor fusion algorithm which overlays point cloud with spectral map</p>
<p>Desktop app features UI for sensor data capture, processing and editing, hardware neural network data analysis and visualisation, dataset cloud storage and collaborative editing</p>
<p>Frameworks: OpenCV, PyTorch3D, ROS</p>
<p>Configuration design, framework stack and methodology suggestions are welcome</p>
<p>Reach out to <a href="mailto:southsidewrongside066@gmail.com">southsidewrongside066@gmail.com</a> for more details</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/2/e2fc6f31185201f42b2fe2e88a19c41b394408af.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e2fc6f31185201f42b2fe2e88a19c41b394408af" title="photon eng.010"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2fc6f31185201f42b2fe2e88a19c41b394408af_2_690x388.jpeg" alt="photon eng.010" data-base62-sha1="wo0KhbDuV2waXnKB8erUHNPl65V" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2fc6f31185201f42b2fe2e88a19c41b394408af_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2fc6f31185201f42b2fe2e88a19c41b394408af_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/2/e2fc6f31185201f42b2fe2e88a19c41b394408af_2_1380x776.jpeg 2x" data-dominant-color="EEEFF1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">photon eng.010</span><span class="informations">1920×1080 103 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Based on the description it seems you are targeting TF, so I would recommend to post the job offer in their discussion board <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:"></p>

username: Nikita_Mezh
cooked: <p>Oh, thanks for your attention! Actually wanted to explore capabilities of both frameworks, as i see, PyTorch has a library for spatial data analysis</p>
Pytorch Opencl Support

username: BestITUserEUW
cooked: <p>Hey,<br>
I am Currently using a Voxl from the Company Modal AI and id like to run a Pytorch Model on Gpu. It doesn’t work with Cuda because it’s a Qualcomm Gpu and Cuda only Supports Nvidia Gpus. So Modal AI is giving a image with Linux Yocto Jethro os and docker ,opencl preinstalled. Does Pytorch Support Opencl?.(Build Specs are listed below)</p>
<p>Board Specs:</p>
<ul>
<li>Qualcomm Snapdragon 821 w/ 4GB LPDDR4 1866MHz</li>
<li>Quad-core CPU up to 2.15GHz</li>
<li>Adreno 530 GPU at 624MH</li>
</ul>
<p>OS: Linux Yocto Jethro with 3.18 kernel</p>
How to inference under DDP

username: mangoxb
cooked: <p>How can I inference model under distributed data parallel?<br>
I want to gather all predictions to calculate metrics and write result in one file.</p>

username: rvarm1
cooked: <p>Hi, At a high level, after training your model with DDP, you can save its <code>state_dict</code> to a path and load a local model from that state_dict using <code>load_state_dict</code> . You can find full documentation here: <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html#save-and-load-checkpoints</a>.</p>

username: Rakshith_V
cooked: <p>I had a similar issue,<br>
Now i’m able to load the model but I got this error</p>
<p>AttributeError: ‘DistributedDataParallel’ object has no attribute ‘generate’</p>
<p>Any thoughts on how to solve this ?</p>

username: pbelevich
cooked: <p>Show you code please and pytorch version</p>

username: Rakshith_V
cooked: <p>Hey, I found a way</p>
<p>When I wrap the model to DDP , all my original model attributes can be found in model.module (where model is equivalent to DDP model)</p>
Not fully connected layer

username: papyrus
cooked: <p>Hi everyone,</p>
<p>I would like to implement a layer looking like a fully connected one, but that has less connections. I managed to it (I think), but it seems like it is really slow. Is it possible to do this in a faster way?</p>
<p>Here is the code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn


if __name__ == '__main__':
    # random tensor
    x = torch.randn(1, 1, 1024)

    # create linear layers
    size_window = 16

    nb_linear_needed = 1024 // size_window

    linear_layers = nn.ModuleList()

    for i in range(nb_linear_needed):
        linear_layers.append(nn.Linear(size_window, size_window // 2))

    # pass every part of x through all linear layers
    new_x = torch.zeros(x.shape[0], x.shape[1], x.shape[2] // 2).cuda()

    for i, linear_layer in enumerate(linear_layers):
        new_borne_inf = i * (size_window // 2)
        new_borne_sup = new_borne_inf + size_window // 2

        borne_inf = i * size_window
        borne_sup = borne_inf + size_window

        new_x[:, :, new_borne_inf:new_borne_sup] = linear_layer(x[:, :, borne_inf:borne_sup])

    print('x shape:', x.shape)
    print('new_x shape:', new_x.shape)

    exit()
</code></pre>
<p>Thanks!</p>

username: my3bikaht
cooked: <p>There was an implementation of sparse linear by huggingface: <a href="https://github.com/huggingface/pytorch_block_sparse" rel="noopener nofollow ugc">https://github.com/huggingface/pytorch_block_sparse</a><br>
But even with using Cutlass it loses to regular fully connected layer due to efficient optimization of the latter.</p>

username: papyrus
cooked: <p>Thanks for the resource, but is there a way to control which part of the matrix is set to zero  with this method?</p>
<p>Alternatively, I found this: <a href="https://stackoverflow.com/questions/70269663/how-to-efficiently-implement-a-non-fully-connected-linear-layer-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">python - How to efficiently implement a non-fully connected Linear Layer in PyTorch? - Stack Overflow</a></p>
Re-train a pre-trained model

username: Mohamed_Nabih
cooked: <p>I have a pre-trained model for speech enhancement. I have already uploaded it and resumed the training from this checkpoint (i.e. set the model to model. train ()) on the same dataset. But while resuming the training I didn’t update its loss i.e. no loss backward, because I think it already sees the data before and doesn’t need to update its parameter. <strong>The problem</strong> is that I see its loss starts to diverge. Is that normal?</p>

username: ptrblck
cooked: <p>I’m not sure what “loss starts to diverge” means, but note that <code>model.train()</code> will e.g. update the running stats of batchnorm layers and will also enable dropout, which might change the model performance during deployment.</p>

username: Mohamed_Nabih
cooked: <p>I mean that the loss value increases, not decreases.</p>
How to BP loss gradient to only a subset of network parameters?

username: Dazitu616
cooked: <p>I’m training a network (say it’s consisted of <code>module1</code> and <code>module2</code>) with two losses <code>loss1</code> and <code>loss2</code>. For <code>loss1</code>, I only want to BP it to <code>module1</code>, while for <code>loss2</code> I want to BP it to the whole network. Can I do something like this to achieve so? Is there any more elegant way to do so?</p>
<pre><code class="lang-auto">optimizer.zero_grad()

loss1.backward(retain_graph=True)  # first BP loss1
model.module2.zero_grad()  # zero out grad for module2, so that only module1 is affected by loss1

loss2.backward()  # loss2 BP to both modules
optimizer.step()
</code></pre>
<p>My second question is that, apart from <code>module.zero_grad()</code>, can I use <code>param.grad.data.zero_()</code> for some individual params which don’t belong to any modules?</p>
<p>And my final question is how to extend this when using amp training? Will the following code work? (My main concern is that do I need to do explicit <code>scaler.unscale_(optimizer)</code> before performing <code>module2.zero_grad()</code>)</p>
<pre><code class="lang-auto">optimizer.zero_grad()

with torch.cuda.amp.autocast():
    loss1, loss2 = calculate_loss(xxx)

scaler.scale(loss1).backward(retain_graph=True)
model.module2.zero_grad()

scaler.scale(loss2).backward()
scaler.step(optimizer)
scaler.update()
</code></pre>

username: ptrblck
cooked: <p>Instead of zeroing out some gradients, you could also pass the <code>inputs</code> argument to <code>backward</code> to calculate only the gradients for these tensors.<br>
Yes, I don’t see any issue with using the approach in <code>amp</code> (once you’ve verified it’s working without <code>amp</code>).</p>

username: Dazitu616
cooked: <p>Thank you! It seems working as expected now.</p>
PyTorch pip installation not working

username: TrippelZ
cooked: <p>Trying to install PyTorch with the given command from the homepage with Stable, Windows, Pip, Python and Cuda 11.3.</p>
<p>The command I get is<br>
<code>pip3 install torch==1.10.0+cu113 torchvision==0.11.1+cu113 torchaudio===0.10.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html</code></p>
<p>I get the error</p>
<pre><code class="lang-auto">ERROR: Could not find a version that satisfies the requirement torch==1.10.0+cu113 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.10.0+cu113
</code></pre>
<p>I have Python 3.10.0 installed, in 64 bit (same as my system) with Pip installed with it.<br>
I have CUDA 11.5 installed from NVIDIA’s site.</p>
<p>I have looked multiple places and found no solution, please help.</p>

username: suegi
cooked: <p>The issue is with your python version. You need python 3.8 because the repositories aren’t yet available for python 3.10</p>

username: TrippelZ
cooked: <p>Ah, alright. I didn’t know this. Will try again with Python 3.8. Thanks.</p>

username: sohani
cooked: <p>I am using Python 3.8 with CUDA 11.5 and still get the same error.</p>

username: ptrblck
cooked: <p>Try to update <code>pip</code> and rerun the command.</p>

username: deeplearner20
cooked: <p>I tried upgrading pip and installing with python 3.7. But it did not work.</p>

username: ptrblck
cooked: <p>What kind of error are you seeing?</p>

username: veritas
cooked: <p>PyTorch 1.10 does not support CUDA 11.5. Lower your CUDA to 11.3.1</p>

username: ptrblck
cooked: <p>PyTorch <code>1.10</code> should support CUDA 11.5 in source builds. While the binaries are built with 10.2 and 11.3, the nightlies are also built with 11.5.<br>
The local CUDA toolkit won’t be used unless you are building PyTorch from source or a custom CUDA extension, so <a class="mention" href="/u/sohani">@sohani</a> wouldn’t need to downgrade the local CUDA toolkit unless one of the previous use cases is used.</p>
Testing the model for quality are very bad

username: abraziv1991
cooked: <p>Hi. I’m new to torch, I’m having trouble testing the model for quality, the results are very bad (50/50, 45/55, etc). I can’t find the error, I’ve been struggling for a month now, nothing works. A similar model on TensorFlow works great. Can somebody help me? Maybe I made a mistake in the description of the model or something</p>

username: abraziv1991
cooked: <p>Nobody wants to help?</p>
<p>[image]</p>

username: ptrblck
cooked: <p>Usually general questions such as “my model doesn’t train” are hard to answer as no background was given.<br>
One approach I would use is to try to overfit a small dataset first (e.g. just 10 samples) and make sure the model is able to learn these samples. If not, you might be using a wrong criterion, unnecessary activation functions in the model output, might be accidentally detaching the computation graph etc.</p>

username: abraziv1991
cooked: <p>Thanks for the answer. The background is simple. I am using Python Tensorflow to create and train my model. Got amazing results! The application I’m using needs a significant speed boost, so I decided to rewrite everything in C++. It seemed to me that the most suitable and friendly C ++ API in LibTorch, so I rewrote everything for LibTorch. I did it for the first time, but I studied a lot of information on the Internet. So I can assume that I’m doing everything right. Just checked the experiment, trained the Python TF model and C ++ LibTorch on the same dataset, TF excellent results, Torch does not work, as if the network had not been trained, and it works on the training set. I do not understand what’s the matter. It is unlikely that the matter is in the functions used, because it would not work in TF either, where there is a fundamental error associated with the Torch feature. Can I provide the model description code here? Maybe it’s a matter of data transfer, perhaps somewhere that happens with the type or something like that.</p>

username: abraziv1991
cooked: <p>I solved the problem myself, thanks.</p>
Torch.nn.modules.rnn

username: Digvijay_mishra
cooked: <p>for my project work, I had to go through the documentation of <strong>TORCH.NN.MODULES.RNN</strong> and there I wanted to know exactly which relu activation function is used because I am building RNN from scratch using trained weights and biases. so my self coded models answers were matching with answers of TORCH.NN.MODULES.RNN when I used ‘Tanh’ activation function but when I am using ‘ReLU’ activation function from nn.RELU, my answers are not matching at all and I tried a lot to run given libraries into google collab so that I can also use the same code and get my problem solved but I couldn’t find any clue which can give me solution even after even after installing necessary libraries like pydot. more precisely I want to run that <strong>_VF.rnn_relu</strong> line.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a65f5a3c86aef9b361ead96dc66d037e2785e47d.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a65f5a3c86aef9b361ead96dc66d037e2785e47d" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a65f5a3c86aef9b361ead96dc66d037e2785e47d_2_690x427.jpeg" alt="1" data-base62-sha1="nJNydkcaFMdtJBoLf29lZGRnoND" width="690" height="427" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/6/a65f5a3c86aef9b361ead96dc66d037e2785e47d_2_690x427.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a65f5a3c86aef9b361ead96dc66d037e2785e47d.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/6/a65f5a3c86aef9b361ead96dc66d037e2785e47d.jpeg 2x" data-dominant-color="F1EAE9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">970×601 84.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Best way to feed 2D tensors of varying size from 100x512 to 800x512 with a median size of 1200x512 to a vision transformer than requires to have all the inputs of the same dimension in the same batch

username: Mona_Jalal
cooked: <p>I have input images that are ranging anywhere from 100 to 8000 of 512x512 pixel squares. So, I feed these 512x512 patches into a resnet18 pretrained on imagenet and receive a 1D 512 tensor. Eventually, I concatenate these 1D tensors and end up with a Nx512 tensor (in an offline fashion).</p>
<p>Eventually, my images are converted to Nx512 tensors where N ranges from 100 to 8000 with a median of 1200 patches inside one image.</p>
<p>Does it even make sense to use something like this to pass these intermediate representations to the vision transformer (that I have dropped its embedding and positional encoding) since it requires me to have tensors of the same dimension for all the items in the same batch? How else I could handle this problem?</p>
<p>What I am doing in below code is to check if the first dimension (# patches in an image) is smaller than median of size of patches in an image across all images (1200 here), then zero pad that Nx512 tensor to convert it to 1200x512. However, if the intermediate representation has more than 1200 patches, I sample 1200 patches randomly from it. This assures me that inside one batch (e.g. of size 32), all the Nx512 tensors are of the same shape of 1200x512.</p>
<p>My main concern is both of these methods, especially the zero-filling are aggressive and I am concerned if they are contributing to my low-accuracy reported in <a href="https://discuss.pytorch.org/t/poor-predictions-for-labels-in-binary-classification-using-cross-entropy-loss/143156" class="inline-onebox">Poor predictions for labels in binary classification using cross-entropy loss</a></p>
<pre><code class="lang-auto">        if features.shape[0] &lt;= 1200:
            a = torch.zeros((1200 - features.shape[0], 512)) #zero padding to lenght 1200
            embeddings = torch.cat((features, a), axis=0)
            sample['image'] = embeddings
        else: 
            random_indices = torch.randint(features.shape[0], (1200, )) 
            sample['image'] = features[random_indices, :]
</code></pre>
Down-resing a 3D tensor

username: Omroth
cooked: <p>I have a tensor of shape [5<em>sx,5</em>sy,5*sz] and I’d like to create a tensor with shape [sx,sy,sz] where the value of each element:</p>
<p>new_tensor[x,y,z] = mean(old_tensor[5x:5x+5,5y:5y+5,5z:5z+5])</p>
<p>Preferably in one instruction that’s fast (and followed by autograd)</p>
<p>Any ideas?<br>
Thanks,<br>
Ian</p>

username: Omroth
cooked: <p>kernel_size=[5,5,5]<br>
torch.nn.Conv3d(in_channels=1,out_channels=1,kernel_size=kernel_size, stride=kernel_size, padding=0)</p>

username: KFrank
cooked: <p>Hi Ian!</p>
<aside class="quote no-group" data-username="Omroth" data-post="2" data-topic="143081" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/fbc32d/48.png" class="avatar"> Omroth:</div>
<blockquote>
<p>kernel_size=[5,5,5]<br>
torch.nn.Conv3d(in_channels=1,out_channels=1,kernel_size=kernel_size, stride=kernel_size, padding=0)</p>
</blockquote>
</aside>
<p>You might prefer to use <code>AvgPool3d</code>.  You can view it as a special<br>
case of <code>Conv3d</code>:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.10.2'
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt; t = torch.randn (5, 10, 15, requires_grad = True)
&gt;&gt;&gt; torch.nn.AvgPool3d (5) (t.unsqueeze (0).unsqueeze (0)).squeeze (0).squeeze (0)
tensor([[[-0.2630, -0.0570,  0.1348],
         [ 0.2163,  0.0359,  0.0404]]], grad_fn=&lt;SqueezeBackward1&gt;)
&gt;&gt;&gt; conv = torch.nn.Conv3d (1, 1, 5, stride = 5, bias = False)
&gt;&gt;&gt; conv.weight.requires_grad = False
&gt;&gt;&gt; _ = conv.weight.copy_ (torch.ones (1, 1, 5, 5, 5) / 125)
&gt;&gt;&gt; conv (t.unsqueeze (0).unsqueeze (0)).squeeze (0).squeeze (0)
tensor([[[-0.2630, -0.0570,  0.1348],
         [ 0.2163,  0.0359,  0.0404]]], grad_fn=&lt;SqueezeBackward1&gt;)
</code></pre>
<p>I haven’t done any timings, but I would expect <code>AvgPool3d</code> to be<br>
modestly faster.</p>
<p>And stylistically, <code>AvgPool3d</code> – at least for me – more clearly “says<br>
what you mean.”</p>
<p>Best.</p>
<p>K. Frank</p>
IndexError: arrays used as indices must be of integer (or boolean) type

username: praggya
cooked: <p>“”"<br>
def _resample_data(X, y, N):</p>
<pre><code>    """Limit sampling to N instances per class."""

    if N &gt; 0:

        # Split labels into set of indexes for each class

        class_idxs = [np.where(y == c)[0] for c in np.unique(y)]

        # Shuffle each of sets of indexes

        [np.random.shuffle(i) for i in class_idxs]

        # Take N indexes, or fewer if total is less than N

        subset_idx = [i[:N] if len(i) &gt;= N else i for i in class_idxs]

        # Use advanced indexing to get subsets of X and y

      

        idxs = np.array(subset_idx).ravel()

        np.random.shuffle(idxs)

        X, y = X[idxs], y[idxs]

    return X, y
</code></pre>
<p>X,y = {},{}</p>
<p>tgt_num = 100</p>
<h2>
<a name="xtgt-ytgt-_resample_datatgt_x-tgt_y-tgt_num-usrlocallibpython37dist-packagesipykernel_launcherpy15-visibledeprecationwarning-creating-an-ndarray-from-ragged-nested-sequences-which-is-a-list-or-tuple-of-lists-or-tuples-or-ndarrays-with-different-lengths-or-shapes-is-deprecated-if-you-meant-to-do-this-you-must-specify-dtypeobject-when-creating-the-ndarray-from-ipykernel-import-kernelapp-as-app-1" class="anchor" href="#xtgt-ytgt-_resample_datatgt_x-tgt_y-tgt_num-usrlocallibpython37dist-packagesipykernel_launcherpy15-visibledeprecationwarning-creating-an-ndarray-from-ragged-nested-sequences-which-is-a-list-or-tuple-of-lists-or-tuples-or-ndarrays-with-different-lengths-or-shapes-is-deprecated-if-you-meant-to-do-this-you-must-specify-dtypeobject-when-creating-the-ndarray-from-ipykernel-import-kernelapp-as-app-1"></a>X[‘tgt’], y[‘tgt’] = _resample_data(tgt_X, tgt_y, tgt_num)<br>
“”"<br>
/usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:15: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify ‘dtype=object’ when creating the ndarray<br>
from ipykernel import kernelapp as app</h2>
<p>IndexError                                Traceback (most recent call last)<br>
 in ()<br>
1 X,y = {},{}<br>
2 tgt_num = 100<br>
----&gt; 3 X[‘tgt’], y[‘tgt’] = _resample_data(tgt_X, tgt_y, tgt_num)</p>
<p> in _resample_data(X, y, N)<br>
15            idxs = np.array(subset_idx).ravel()<br>
16            np.random.shuffle(idxs)<br>
—&gt; 17            X, y = X[idxs], y[idxs]<br>
18<br>
19        return X, y</p>
<p>IndexError: arrays used as indices must be of integer (or boolean) type</p>

username: praggya
cooked: <p>Please help anyone urgently!!!</p>

username: thecho7
cooked: <ol>
<li>Check whether <code>class_idxs</code> is empty or not</li>
<li>If <code>class_idxs</code> is not empty, type casting is probably what you want</li>
</ol>
<pre><code class="lang-auto">idx = idx.astype(np.int64)
</code></pre>
<p>I think your question is kinda closer to <strong>Stackoverflow</strong> instead of here.<br>
Anyway, I hope you can solve</p>
Fast computation of the Hessian diagonal

username: amerlo94
cooked: <p>Hi,</p>
<p>In my use case, I need to compute the diagonal of the Hessian for 3D tensors.</p>
<p>The code below illustrates what I need to compute. It works, however, it does not scale well if one increases the dimension of the tensor.</p>
<p>Are there faster way to achieve the same computation?</p>
<p>I am on <code>torch=1.10.0</code>.</p>
<pre><code class="lang-auto">from time import time

import torch
from torch._vmap_internals import _vmap as vmap

def grad(outputs, inputs, **kwargs):
    """
    Compute the trace of the jacobian of a 3D tensor.

    grad_{ijk} = d (outputs_{ijk}) / d (inputs_{ijk})

    See: https://discuss.pytorch.org/t/jacobian-functional-api-batch-respecting-jacobian/84571/7?u=amerlo94
    """

    shape = outputs.shape
    bs = shape[0]
    n = shape[1] * shape[2]

    outputs = outputs.view(bs, n)
    outputs = outputs.sum(axis=0)
    grad_outputs = torch.eye(n, dtype=outputs.dtype, device=outputs.device)

    def get_vjp(v):
        return torch.autograd.grad(outputs, inputs, v, **kwargs)[0]

    vjp = vmap(get_vjp)
    grad = vjp(grad_outputs)

    return grad.T.view(shape)


n = 16
b = torch.randn(1, n, n)


def f(x):
    return x ** 2 * b


bs = 128
iterations = 1000

x = torch.rand(bs).requires_grad_(True)
x = x.view(-1, 1, 1)
y = f(x)

#  Check gradients
jac = grad(y, x, create_graph=True)
hes = grad(jac, x)
assert torch.allclose(jac, 2 * x * b)
assert torch.allclose(hes, 2 * b)

#  Time gradients
t0 = time()
for _ in range(iterations):
    jac = grad(y, x, create_graph=True)
    grad(jac, x)

tottime = time() - t0

time_per_batch = tottime / iterations / bs
print(f"Time per batch: {time_per_batch * 1e6:.2f} us")
</code></pre>
Torch.repeat_interleave on dictionary?

username: pineapple_pine
cooked: <p>Hi everyone! I need to repeat an object ( in type dictionary). Wanna use the torch.repeat_interleave function. What’s the best way of doing so? Shall I convert the dictionary or list to tensor first?</p>
<p>Thank you!</p>

username: ptrblck
cooked: <p>Yes, <code>torch.repeat_interleave</code> expects a tensor as its <code>input</code>.</p>

username: pineapple_pine
cooked: <p>Thank you! Is there a function that directly do so, or do I need to write a for-loop for the conversion?</p>

username: ptrblck
cooked: <p>If you’ve created a <code>list</code> containing tensors with the same shape you could use e.g. <code>torch.stack</code> or <code>torch.cat</code>:</p>
<pre><code class="lang-python">l = [torch.randn(10, 10) for _ in range(3)]

x = torch.stack(l)
print(x.shape)
# &gt; torch.Size([3, 10, 10])

y = torch.cat(l)
print(y.shape)
# &gt; torch.Size([30, 10])
</code></pre>
<p>For more advanced use cases it would depend on the actual use case.</p>

username: pineapple_pine
cooked: <p>Thank you very much!</p>
CMake with Pytorch 1.10.2 no CUDA

username: gmguarino
cooked: <p>Hi,</p>
<p>I am trying to compile a test program with Pytorch in C++. Ideally I would like to load a model and perform inference using C++. I downloaded libtorch 1.10.2 with no GPU (I double checked it), however I get an error as there is no CUDA toolkit.</p>
<p>I am developing on an old Macbook Pro 9.2 running Linux Mint 20.2 x86_64.</p>
<p>Has anyone solved this?<br>
Giuseppe</p>
<p>My project dir:</p>
<pre><code class="lang-auto">inference
|-inference.cpp
|-CMakeLists.txt
|-build/
|-libtorch/
</code></pre>
<p>I am running this command:<br>
<code>cmake -DCMAKE_PREFIX_PATH=/home/gmguarino/Documenti/Exercises/C++/inference/libtorch ..</code></p>
<p>And getting the following error:</p>
<pre><code class="lang-auto">CUDA_TOOLKIT_ROOT_DIR not found or specified
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS CUDA_CUDART_LIBRARY) 
CMake Warning at .venv/lib/python3.8/site-packages/torch/share/cmake/Caffe2/public/cuda.cmake:31 (message):
  Caffe2: CUDA cannot be found.  Depending on whether you are building Caffe2
  or a Caffe2 dependent library, the next warning / error will give you more
  info.
Call Stack (most recent call first):
  .venv/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:88 (include)
  libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:17 (find_package)


CMake Error at .venv/lib/python3.8/site-packages/torch/share/cmake/Caffe2/Caffe2Config.cmake:90 (message):
  Your installed Caffe2 version uses CUDA but I cannot find the CUDA
  libraries.  Please set the proper CUDA prefixes and / or install CUDA.
Call Stack (most recent call first):
  libtorch/share/cmake/Torch/TorchConfig.cmake:68 (find_package)
  CMakeLists.txt:17 (find_package)
</code></pre>
Torchtext/API documentation of Vectors/stoi

username: serge1
cooked: <p>Hello, I’m using pytorch to teach NLP, and I would like to avoid giving bad habits to my students.<br>
An exercice I give involves the usual manipulations of GloVe vectors (computing nearest neighbourg and so on). After the change of API for torchtext, I’m a bit nervous about what I should use.</p>
<p>The <a href="https://pytorch.org/text/stable/vocab.html#vectors" rel="noopener nofollow ugc">current documentation</a> lists very few methods and variables ; calling help() on Vectors or on GloVe is a bit more informative, but, for instance, fields <code>vectors</code> or <code>itos</code> are not documented any more (or I miss something). It might come from my strongly typed languages background, but I like to know what’s ok for use and what could change - basically, the recommended way to use the classes.</p>
<p>Should I relax, and given that my code works, teach the use of itos to my students, or do I miss something ?</p>
<p>Best Regards,</p>
<p>Serge</p>
Inference pipelines examples

username: saeid93
cooked: <p>Hi community<br>
I am doing research about autoscaling of ML inference pipelines. Currently I am looking for examples, repos or datasets of real-world examples of ML inference pipelines, I was wondering if anyone could provide me real-world examples of inference pipelines being used in production or are used as benchmarks? e.g. for microservices <a href="https://github.com/clowee/MicroserviceDataset" rel="noopener nofollow ugc">GitHub - clowee/MicroserviceDataset: Microservice Dependency Graph Dataset</a> there is this dataset of microservices graphs with the links to the repos but I wasn’t able to find something similar for the ml inference graphs. There is this work that is based on some Google dataset training pipelines <a href="https://arxiv.org/abs/2103.16007" rel="noopener nofollow ugc">[2103.16007] Production Machine Learning Pipelines: Empirical Analysis and Optimization Opportunities</a> but unfortunately it is not open source. whether if you know some examples of real-world repos or a metadata dataset of the graphs both will be super helpful to me.</p>