akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How can i remove layers from torch_model?

username: Matheus_Silva_de_Pau
cooked: <p>I’m trying to remove layers from this model and at the same time keep the “TorchModel”, how can I do that? I tried using .children, but a problem occurs in another line of code.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97494d7fea78b7aedb5ee066ed5345e4c210417d.png" data-download-href="https://discuss.pytorch.org/uploads/default/97494d7fea78b7aedb5ee066ed5345e4c210417d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/97494d7fea78b7aedb5ee066ed5345e4c210417d.png" alt="image" data-base62-sha1="lAl9cwrc093T55U0Od2D5eey5Q1" width="690" height="430" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">783×489 19.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: thecho7
cooked: <p>I don’t know the direct method to modify the name of class. Instead, use <code>.named_children()</code></p>
<pre><code class="lang-auto">class TorchModel(nn.Module):
    def __init__(self, torch_model):
        super(TorchModel, self).__init__()
        for n, m in torch_model.named_children():
            self.__setattr__(n, m)

    def forward(self, x):
        # whatever you want

my_model = TorchModel(torch_model)
</code></pre>
Pass variables to autorgrad to implement dynamic regularization

username: haritsahm
cooked: <p>Hi guys. I want to implement a dynamic regularization proposed in “Convolutional Neural Networks with Dynamic Regularization”. I’m new to custom forward and backward pass using autograd and I found  code references from <a href="https://github.com/rra94/ShakeShake2.0/blob/master/models/shakeshake.py" rel="noopener nofollow ugc">Shake-Shake</a> and <a href="https://github.com/Dragonsson/SakeDrop-Pytorch/blob/master/models/shakedrop.py" rel="noopener nofollow ugc">Shake-Drop</a> methods mentioned by the author.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/347a5842290cad35d6b7bb67088d7fb95ece0ad2.png" data-download-href="https://discuss.pytorch.org/uploads/default/347a5842290cad35d6b7bb67088d7fb95ece0ad2" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/347a5842290cad35d6b7bb67088d7fb95ece0ad2_2_305x250.png" alt="image" data-base62-sha1="7ueWXgvDa99lB1HLQqdAAPgVUFI" width="305" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/347a5842290cad35d6b7bb67088d7fb95ece0ad2_2_305x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/347a5842290cad35d6b7bb67088d7fb95ece0ad2_2_457x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/347a5842290cad35d6b7bb67088d7fb95ece0ad2_2_610x500.png 2x" data-dominant-color="F5F6F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">629×514 32.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To implement the dynamic regularization, the difference between the training losses is passed to the forward and backward training. My question is, how do I pass this variable to autograd function? Both previous methods and autograd examples only use the grad_output, but the proposed method pass other values as well.</p>
Inference extremely slow on AWS (CPU)?

username: alsuhr
cooked: <p>Hi,</p>
<p>I’m trying to run inference on a trained model using an AWS EC2 instance (specifically, the c5 series) using CPUs for inference. However, I noticed that the forward pass is extremely slow on CPU when compared to even just running inference on my laptop (2018 Macbook Pro, also using CPU). E.g., the entire forward pass takes 0.05 s on my laptop, but a single line in the forward pass (a call to conv2d) takes 0.15 s on the AWS instance.</p>
<p>I’m using pytorch 1.3.1 and based my AMI on the Deep Learning AMI.</p>
<p>Any ideas why it is so slow, or what I could do to speed it up? (Or, if I should just use an instance with GPUs?)</p>
<p>Edit: I tried timing with and without MKL (like here: <a href="https://discuss.pytorch.org/t/use-mkldnn-in-pytorch/54943/4" class="inline-onebox">Use MKLDNN in pytorch - #4 by LeviViana</a>), and with MKL was actually <em>slower</em> than without it.</p>
<p>Thanks!</p>

username: marksaroufim
cooked: <p>Yeah so there’s a lot of things you can do to make CPU inference faster, sorted by increasing complexity</p>
<ol>
<li>Use an m6i or c6i instance they are muuuch faster</li>
<li>Set <code>torch.set_num_threads(1)</code> in your script</li>
<li>Use Intel IPEX</li>
<li>Use Intel CPU launcher script</li>
</ol>
<p>Using these kinds of tricks I managed to get BERT inference from 2s to roughly 20ms</p>
<p>3 -4 are integrated in torchserve if you’re interested <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>

username: Jiong_Gong
cooked: <p>You can get more information about IPEX from the github link below. IPEX currently works with PyTorch 1.10 release.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/intel/intel-extension-for-pytorch">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/intel/intel-extension-for-pytorch" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/47485eee40f78d7b2f444357c732a2b2cb1a0de9669d22b1de92454d4e8cae12/intel/intel-extension-for-pytorch" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/intel/intel-extension-for-pytorch" target="_blank" rel="noopener nofollow ugc">GitHub - intel/intel-extension-for-pytorch: A Python package for extending...</a></h3>

  <p>A Python package for extending the official PyTorch that can easily obtain performance on Intel platform - GitHub - intel/intel-extension-for-pytorch: A Python package for extending the official Py...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And here is the documentation site of it:<br>
<a href="https://intel.github.io/intel-extension-for-pytorch/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://intel.github.io/intel-extension-for-pytorch/</a></p>
Giant tensor consumes GPU memory

username: Yunchao_Liu
cooked: <p>Hello,</p>
<p>I’m tring to calculate the cosine similarity between two tensors of different shapes:<br>
tensor1: torch.Size([15, 24, 4, 120])<br>
tensor2: torch.Size([5608, 4, 120])</p>
<p>To make them have the same dimension, I used unsqueeze to make them both of size<br>
torch.Size([15, 24, 5608, 4, 120])</p>
<p>However, this new tensor makes me run out of memory because it has 969M elements and takes 3.8Gb.</p>
<p>Is there a better way to calculate cosine similarity without creating this giant tensor?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/yunchao_liu">@Yunchao_Liu</a></p>
<p>you can try something like</p>
<pre><code class="lang-auto">tensor1 = torch.rand([15, 24, 4, 120])
tensor2 = torch.rand([5608, 4, 120])
cos = nn.CosineSimilarity(dim=0)

def custom_cos(cur_tensor):
    return cos(cur_tensor, tensor2)

results = map(custom_cos, tensor1.view(-1, 4, 120))
results = torch.concat(list(results)).view(15,24,4,120)
results.size()
</code></pre>
<p>If you have the latest version you can also use<br>
<a href="https://pytorch.org/tutorials/prototype/vmap_recipe.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/prototype/vmap_recipe.html</a></p>

username: Yunchao_Liu
cooked: <p>Thanks <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> ! I will give it a try.</p>

username: Yunchao_Liu
cooked: <p>Hi <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a> . I tried the method but it is too slow. I need to compute this tensor in every iteration of my forward function. So the speed is also an issue.</p>
TorchServe: Access different models with "--models all" option

username: jbm
cooked: <p>I’d like to create a TorchServe handler that draws on 3 models, but I’m not clear on how to access/identify the different models in the script, when starting TorchServe with the <code>--models all</code> option.</p>
<p>Any help appreciated.</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/jbm">@jbm</a> we built our workflow feature to handle such use cases, let me know if this works for you but you’ll be able to define how the 3 models interact in a YAML script  <a href="https://github.com/pytorch/serve/tree/master/examples/Workflows" rel="noopener nofollow ugc">https://github.com/pytorch/serve/tree/master/examples/Workflows</a></p>

username: jbm
cooked: <p>Ah, okay, I’ll have a look. Thanks!</p>
How to sync Optimizer parameters during DDP training

username: swenson-nick
cooked: <p>Hi,</p>
<p>I am writing a customer optimizer that needs to be able to access the global gradient from the previous iteration, as well as sync parameters across optimizer instances.</p>
<p>What would be the best way to do this?</p>
<p>In a non-distributed context, I simply stored the grads in the state variable, as ADAM does with state[‘exp_avg_sq’].</p>
<p>Should I be calling my own all_reduce during the training loop?</p>
<p>Thanks!</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/swenson-nick">@swenson-nick</a>, DDP will guarantee that all ranks sees the same <code>param.grad</code> values (by automatically running allreduce during backward) before getting into <code>optimizer.step()</code>. Will that be sufficient? Or do you have to read the local gradients?</p>
Question about pytorch inplace operation

username: ZiQing_Li
cooked: <ol>
<li>Can you list the inplace operations in Pytorch?</li>
<li>Whether the inplace operations are encouraged in Pytorch?</li>
<li>How to avoid using inplace operations which I may not know?</li>
</ol>

username: ptrblck
cooked: <ol>
<li>All inplace operations have a trailing underscore (e.g. <code>tensor.add_</code>, <code>tensor.mul_</code>), are using the “inplace syntax” (e.g. <code>tensor += 1</code>), or are directly accessing the data (e.g. <code>tensor[0] = 1.</code>).</li>
<li>While inplace ops can save memory, they could disallow fused ops if your model is scripted. Their usage might depend on your actual use case, but you could stick to the default out-of-place ops.</li>
<li>Don’t use ops mentioned in 1.</li>
</ol>
Question about torch.clone()

username: ZiQing_Li
cooked: <p>As I know, the operation of <code>clone</code> is keeping track of the <strong>original</strong> history of operations and passing the gradient to the original one. So I’m confused at what time should I use <code>clone()</code> because I think that cloned tensor will have the effect (i.e. the gradients pass to the original tensor are equal) as the original one after the net backward?</p>

username: ptrblck
cooked: <p>While <code>.clone()</code> is differentiable and the gradient will thus be forwarded to the original tensor, e.g. inplace operations won’t change the original tensor.</p>
Which Machine for QAT using QNNPACK backend?

username: dalseeroh
cooked: <p>Hello folks,</p>
<p>I wrote a UNet-like model recently, and I am trying to deploy it to Android device(ARMv8.2). Therefore, the model needs to be quantized beforehand, and I am thinking of QAT. After spending some time for research, I found that I cannot run this command on my Windows machine, which has Intel CPU.</p>
<pre><code class="lang-auto">model_fp32.qconfig = torch.quantization.get_default_qconfig('qnnpack')
</code></pre>
<p>So my question is, where should I run this command? Jetson Nano, Coral Dev board, or whatever boards that have arm cpu can run this, correct? Or if there is any other approach to quantize my model to fit the purpose? Can the cross-compiling do the job for QAT with qnnpack?</p>

username: marksaroufim
cooked: <p>You need to quantize the model on the device you want the model to eventually run. I’d start with Dynamic quantization, it’s easier to get working than QAT</p>
No module named 'torch' when I make exe file using pyinstaller

username: HISTRAVEL_Nicholas
cooked: <p>[Environment]</p>
<ul>
<li>vscode</li>
<li>Anaconda</li>
</ul>
<p>when I make exe file using pyinstaller, I have the below error message even if I successfully ran main_for_exe.py.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53e8d09723ed86f78782aeeb212adcf7ab516a69.png" data-download-href="https://discuss.pytorch.org/uploads/default/53e8d09723ed86f78782aeeb212adcf7ab516a69" title="제목 없음"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/3/53e8d09723ed86f78782aeeb212adcf7ab516a69.png" alt="제목 없음" data-base62-sha1="bYiqSfPD35drYed6ykjxMx4KoIN" width="690" height="126" data-dominant-color="393939"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">제목 없음</span><span class="informations">979×179 5.72 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>There are some solutions: "add module to hiddenimports<br>
but, it does not work.</p>
<p>How do I solve this???</p>

username: marksaroufim
cooked: <p>I haven’t used pyinstaller but I figure it needs some link to a requirements.txt or a link to the pytorch source code</p>
Question about PyTorch 1.8 performance optimization for CPU inference

username: jrandel
cooked: <p>So I have a unique situation where I’ve got access to a large cluster but no GPU. I’m currently running GPT-2 large without issue using CPU inference, the response time is less than two seconds which is acceptable for a conversational agent. GPT-2 xlarge clocks in at 4:30 minutes, so I am curious what options there may be with PyTorch 1.8 for optimizing CPU inference?</p>
<p>Each CPU has six (6) cores, does the stock PyTorch 1.8 distribution include multithreading support for CPU inference?</p>
<p>Are there any distributed inference options available now with this distributed training announcement?</p>
<p>Thanks in advance</p>

username: marksaroufim
cooked: <p>Just answered this here <a href="https://discuss.pytorch.org/t/inference-extremely-slow-on-aws-cpu/111310/2" class="inline-onebox">Inference extremely slow on AWS (CPU)? - #2 by marksaroufim</a></p>
When to convert Python Float to Pytorch Tensor

username: misery_business
cooked: <p>Is there a benefit/downside to converting a Python float to a Pytorch Tensor? Does it make a difference for speed and/or device placement?</p>
<pre><code class="lang-auto">a = torch.Tensor([1.])

# option A 
b = a + 2. 

# option B
c = a + torch.Tensor([2.])
</code></pre>

username: marksaroufim
cooked: <p>Try profiling it, whether it’s useful or not really depends on whether you’re going to do more PyTorch operations on the Python float</p>
Inference of transformer model on mac os x laptop

username: simonjaq
cooked: <p>Hi All</p>
<p>I have been working on an email assistant. I fine tuned one of Huggingface’s transformers on my emails to classify them into different categories. It works really well on my linux ML workstation</p>
<p>But now I’m trying to implement the model to actually work on my emails in Apple Mail and I’m quite lost on how to do this. I managed to convert the model to ONNX.</p>
<p>I know I can run Applescripts on incoming email. So my idea would be to call an AppleScript on every incoming email, that calls a python script, which runs the model and returns probabilities for the trained categories.<br>
But that all seems quite cumbersome and will probably be slow with the model loading on every incoming email.</p>
<p>Does any of you know a simpler way? Or an easy tutorial or example about how to do this?</p>

username: marksaroufim
cooked: <p>I’d suggest you use an inferencing framework like <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>
<p>So you’d setup your model behind a REST API which you would then call from applescript</p>
How to speed up inference time in Windows?

username: ericliii
cooked: <p>Hi All,</p>
<p>I’m working on a classification task. The scenario is I will detect faces from a video, crop the faces, and feed them to a classifier to get face attributes like gender and age. Here are the details:</p>
<p>Framework: PyTorch<br>
Model: Resnet 34<br>
Input size: (x, 3, 200, 200), x is the number of detected faces, usually 1 or 2.<br>
System: Windows<br>
Python version: 3.7.6<br>
GPU: GeForce RTX 2070Ti (8G)</p>
<p>After I finished training, I tested the inference time using test dataset, and got &lt;10ms per image (it would be slow for the first image, like about 30ms, because PyTorch model needs some warm up. After that, the inference time will be stable around 10 ms or less). Btw, there is only one face in each image.</p>
<p>But when I integrated the model into my software, the inference time is around 50ms which is much slower than my expectation. This is how I loaded the model in software:</p>
<pre><code class="lang-auto">model.load_state_dict(torch.load(model_path))
</code></pre>
<p>I’m sure I moved my input and model to GPU and switched model to eval mode. And my software is based on Python.</p>
<p>Does anyone have idea about this issue?</p>
<p>Thanks, guys!</p>

username: marksaroufim
cooked: <p>I’d check a few simple things like</p>
<ol>
<li>Are you sure the model is running on GPU?</li>
<li>Did you try <code>torch.set_num_threads(1)</code>
</li>
</ol>
<p>And if all of the above fails that’s the benefit of using an inferencing framework like torchserve <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">https://github.com/pytorch/serve</a></p>
Standard way of combine the output of 2 different model

username: youwillknovv
cooked: <p>Hi, I have a dataset which consist of image and caption, I ran 2 pretrained model on image and caption seperately(resnet 50 for image and Albert for caption.)<br>
As the 2 model’s output are the same(18 labels, using sigmoid) I am currently adding the output of 2 models together and divide by 2. I know this might be a dumb way of doing it. Can anyone give me some advice on what is the standard way of combining the output of 2 models? Thanks!</p>

username: marksaroufim
cooked: <p>Do you have labels for the combined problem, if so pick a linear weight for each output <code>w_1 * m_1 + w_2 * m2</code> and learn <code>w_1</code> and <code>w_2</code> if you don’t adding them or just picking your own weights can often work as a stopgap solution</p>
How to improve model latency for deployment

username: Mohamed_Zahran
cooked: <p><strong>Question:</strong><br>
How to improve model latency for web deployment without retraining the models? What is the checklist that I should mark to improve the model speed?</p>
<p><strong>Context:</strong><br>
I have multiple models that process a video sequentially on one machine with one K80 GPU; each model takes around 5 mins to process a video that is 1 min long. What ideas and suggestions should I try to improve each model latency without changing the model architecture? How should I structure my thinking about this problem?</p>

username: marksaroufim
cooked: <p>So this is an involved question but as a baseline I approach these problems like this</p>
<ol>
<li>Setup a benchmark so you can figure out what to improve</li>
<li>Profile using PyTorch profiler to find bottlenecks or bugs</li>
<li>Isolate source of problem (is it preprocessing, training, something else)</li>
<li>If training then look into smaller models, quantization, larger batch sizes, distillation, pruning - benchmark and see what works best (this is why a benchmark script is important)</li>
<li>If using this for inference use a runtime like IPEX, TensorRT, Torschscript ORT</li>
<li>If none of the above works which it should most times then time to write your own kernels</li>
</ol>
[Torchserve] Download .mar model with AWS credentials

username: simaiden
cooked: <p>I know that it’s possible to load a model in torchserve using the url to the .mar file and torchserve will download the file for me, but how torchserve could download the file if the url is private in AWS bucket? Especially working with kubernetes.</p>
<p>Thanks!</p>

username: marksaroufim
cooked: <p>This feature already exists <a href="https://github.com/pytorch/serve/pull/1027" rel="noopener nofollow ugc">https://github.com/pytorch/serve/pull/1027</a></p>
Cannot import PyTorch in Alpine Docker Container

username: baris
cooked: <p>The MRE below should be enough to explain what I’m trying to do. Google doesn’t help as it seems to be a unique error, which surprised me since Alpine &amp; PyTorch are both fairly popular.</p>
<p>Dockerfile:</p>
<pre><code class="lang-auto">FROM nginx:stable-alpine as production
WORKDIR /app
RUN apk update &amp;&amp; apk add --no-cache python3 bash &amp;&amp; \
  python3 -m ensurepip &amp;&amp; \
  rm -r /usr/lib/python*/ensurepip &amp;&amp; \
  pip3 install --upgrade pip setuptools &amp;&amp; \
  if [ ! -e /usr/bin/pip ]; then ln -s pip3 /usr/bin/pip ; fi &amp;&amp; \
  if [[ ! -e /usr/bin/python ]]; then ln -sf /usr/bin/python3 /usr/bin/python; fi &amp;&amp; \
  rm -r /root/.cache
RUN apk update &amp;&amp; apk add gcc g++ python3-dev musl-dev
RUN pip install torch==1.10.1 --find-links https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>Build &amp; bash into it:</p>
<pre><code class="lang-auto">sudo docker build -t mini .
sudo docker run --name torch-pls -it --entrypoint bash mini
</code></pre>
<p>Launch Python:</p>
<pre><code class="lang-auto">bash-5.1# python
Python 3.9.5 (default, Nov 24 2021, 21:19:13) 
[GCC 10.3.1 20210424] on linux
Type "help", "copyright", "credits" or "license" for more information.
</code></pre>
<p>Import PyTorch and see it fail:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/lib/python3.9/site-packages/torch/__init__.py", line 196, in &lt;module&gt;
    _load_global_deps()
  File "/usr/lib/python3.9/site-packages/torch/__init__.py", line 149, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/usr/lib/python3.9/ctypes/__init__.py", line 374, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: Error relocating /usr/lib/python3.9/site-packages/torch/lib/libgomp-a34b3233.so.1: pthread_attr_setaffinity_np: symbol not found
</code></pre>
<p>Can anyone help?</p>

username: marksaroufim
cooked: <p>I’m not sure but perhaps this author can help you out <a href="https://github.com/petronetto/pytorch-alpine" rel="noopener nofollow ugc">https://github.com/petronetto/pytorch-alpine</a></p>
<p>I’m assuming things work if you a non alpine starting image?</p>
How to debug torchserve pip install?

username: elephantum
cooked: <p>We’re trying to use torchserve (via KFServing).</p>
<p>We prepared .mar with our model and requirements.txt (most notable dependency is transformers)</p>
<p>When we try to run in torchserve logs I see a line “Custom pip package installation failed for ”</p>
<p>When connect to container and execute “pip install -r requirements.txt” all dependencies install just fine.</p>
<p>Question: how do I get more detailed error about pip install from torchserve?</p>

username: marksaroufim
cooked: <p>I need to document this better but the answer is you need to add this line to config.properties <strong><code>install_py_dep_per_model=true</code></strong></p>
GPU support on Raspberry Pi 3 B

username: Ja1Zhou
cooked: <p>I was trying to add GPU support for Raspberry Pi 3 B . I did found a C++ Library for Raspberry pi GPU computation. There are two possible ways now: to customize my c++ extension, or to extend a new dispatcher. Now I do have cross compile tools, yet I was kind of confused what to do next. It would be really nice if anyone has any recommendations!</p>

username: d4l3k
cooked: <p>Are you trying to train or do inference? If you’re doing inference you can get quite far with model quantization + qnnpack. qnnpack is optimized for arm CPUs and gets pretty good performance on the RPi.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/realtime_rpi.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/realtime_rpi.html</a></p>
<p>Adding a new GPU backend to pytorch is possible though quite a bit of work to support and maintain</p>
Difference Between Static and Dynamic Libtorch

username: Mohammadreza_Nazari
cooked: <p>I used the following links to download static and dynamics versions of the libtorch, but both seem the be the same build:<br>
<a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/nightly/cpu/libtorch-shared-with-deps-latest.zip</a><br>
<a href="https://download.pytorch.org/libtorch/nightly/cpu/libtorch-static-with-deps-latest.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/nightly/cpu/libtorch-static-with-deps-latest.zip</a></p>
<p>I was expecting to see libc10.a, libtorch.a, etc. Apparently, there is something wrong in build process of static binaries. I also compared the binaries using “diff” command and both are exactly the same.</p>

username: Mohammadreza_Nazari
cooked: <p>It looks like that the build process does not set BUILD_SHARED_LIBS=OFF<br>
Can someone provide more details?</p>
Example with gRPC

username: juhyung
cooked: <p>I’m looking for serving pytorch model with grpc with python.<br>
But cannot. find any source…</p>
<p>Is there any guide serving pytorch model with grpc?</p>

username: marksaroufim
cooked: <p>Torchserve has gRPC support - let me know if you have any questions about it <a href="https://github.com/pytorch/serve/blob/master/docs/grpc_api.md" rel="noopener nofollow ugc">https://github.com/pytorch/serve/blob/master/docs/grpc_api.md</a></p>
Does PyTorch overprovision VRAM?

username: OverLordGoldDragon
cooked: <p>A model uses 6GB VRAM, total use 7.8/8GB. I double input size, now it uses 6.1GB, 7.9/8GB. This shouldn’t be possible if pytorch uses the minimal it needs each time, and it’s a problem when I can’t reduce VRAM use by using a smaller model. Any remedy?</p>
Pytorch variable explorer

username: sh0416
cooked: <p>Hi,</p>
<p>I want to see the all tensors that I made and the dependency between them.</p>
<p>If it doesn’t exist, I want to implement it.</p>
<p>Is there anyone who have ever seen this functionality?</p>
<p>I imagine that the final result will be something like this.</p>
<pre><code class="lang-auto">a = torch.zeros(1)
b = torch.zeros(1)
c = torch.zeros(1)
d = torch.zeros(1)
e = a + b
e = e + c

inspect_pytorch()
# Allocated tensor:
#  a (shape=[1], dtype=torch.float)
#  b (shape=[1], dtype=torch.float)
#  c (shape=[1], dtype=torch.float)
#  d (shape=[1], dtype=torch.float)
#  tensor#1 (shape=[1], dtype=torch.float) No name for this tensor
#  e (shape=[1], dtype=torch.float)
#
# Dependency:
#  tensor#1 &lt;- (a, b, op:add)
#  e &lt;- (tensor#1, c, op:add)
</code></pre>
<p>I think these kind of information could be helpful.</p>
<p>Thanks,</p>

username: marksaroufim
cooked: <p>You can already do this with some third party libraries like</p>
<pre><code class="lang-auto">from rich import inspect
inspect(a)
inspect(b)
inspect(c)
</code></pre>
Extract quantized parameters for use in custom compute unit

username: pmousoul
cooked: <p>Hello,</p>
<p>I could not find in the related documentation the following key points (for achieving what the topic title says):</p>
<ol>
<li>
<p>How to extract the quantized parameters from a quantized model?</p>
</li>
<li>
<p>How these parameters can be used for integer only computation?</p>
</li>
<li>
<p>What are the transformations required for keeping integer arithmetic when transitioning from one layer to the next one?<br>
E.g. when two consecutive layers have different quantization parameters then the results of the first layer must be adjusted for use in the next layer - what is the adjustment in done in pytorch and how it is   implemented?</p>
</li>
</ol>
<p>Please, provide a simple example of two consecutive quantized layers along with the required arithmetic for integer-only computation.</p>
<p>The goal is to implement this arithmetic using a custom computing machine in order to be able to execute on it models trained and quantized using pytorch.</p>
<p>Thanks in advance,<br>
Panagiotis</p>

username: HDCharles
cooked: <ol>
<li>you can get the parameters from model.state_dict()</li>
<li>They aren’t. The quantized ops use a combination of float and integer arithmetic (the scale quantization parameter is a float after all). A good description of the math can be found here: <a href="https://leimao.github.io/article/Neural-Networks-Quantization/" class="inline-onebox" rel="noopener nofollow ugc">Quantization for Neural Networks - Lei Mao's Log Book</a></li>
</ol>
<blockquote>
<ol start="3">
<li>What are the transformations required for keeping integer arithmetic when transitioning from one layer to the next one?<br>
E.g. when two consecutive layers have different quantization parameters then the results of the first layer must be adjusted for use in the next layer - what is the adjustment in done in pytorch and how it is implemented?</li>
</ol>
</blockquote>
<blockquote>
<p>Please, provide a simple example of two consecutive quantized layers along with the required arithmetic for integer-only computation.</p>
</blockquote>
<p>I believe both these questions will be answered by reading the above link which may clarify some misconceptions about the process. Otherwise the specifics of the implementations of the quantized ops depend on the backend in question. FBGEMM (<a href="https://github.com/pytorch/FBGEMM" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/FBGEMM: FB (Facebook) + GEMM (General Matrix-Matrix Multiplication) - https://code.fb.com/ml-applications/fbgemm/</a>) and QNNPACK (<a href="https://github.com/pytorch/QNNPACK" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/QNNPACK: Quantized Neural Network PACKage - mobile-optimized implementation of quantized neural network operators</a>) handle this differently and you may be able to find better answers by digging into the specific backend you wish to emulate.</p>

username: pmousoul
cooked: <p>Thank you very much for the very informative answer!</p>
Access Dataset from Local Directory in Cluster

username: r112
cooked: <p>Hello,</p>
<p>I have my celeb-a image dataset folder in my home directory, but my neural network is in my lab’s cluster, so when I get an error when I set root_dir equal to the image dataset folder home directory. I am not sure how to move an image dataset over either.</p>
<p>Best,</p>

username: ptrblck
cooked: <p>The question seems to be more setup specific, so I would recommend to check with your system admin how to move files/folders around or mount specific folders into your workspace.</p>
May I know how are PyTorch's unit tests exercised by PyTorch developers in order to verify a PyTorch release

username: Tsz_On_Li
cooked: <p>Hello! I noticed that PyTorch is bundled with a set of unit test suites (under pytorch/test/) and the tests can be excersied by running run_test.py. However, I also noticed that run_test.py accepts multiple flags/parameters (e.g., --pt, --jit) and I would like to know which flags/parameters are usually passed to the test suite for verifying a PyTorch release. Besides, I would like to know whether PyTorch is verified with test suites other than run_test.py and what are the flags/parameters passed to those test suites. Thank you in advance!</p>

username: marksaroufim
cooked: <p>You probably want to study our github action scripts which run CI. No code gets merged unless these all pass <a href="https://github.com/pytorch/pytorch/tree/master/.github/workflows" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/tree/master/.github/workflows</a></p>
Can't convert np.ndarray of type numpy.uint64. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool

username: yhablatou
cooked: <p>can’t convert np.ndarray of type numpy.uint64. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.</p>
<pre><code>for i in range(1, n_episodes + 1):
    obs = env.reset()
    R = 0  # return (sum of rewards)
    t = 0  # time step
    while True:
        # Uncomment to watch the behavior in a GUI window
        # env.render()
        #action = agent.act(torch.FloatTensor(obs))
        obs = np.array(obs , dtype=np.float32)
        
        obs = torch.from_numpy(obs)

        action = agent.act(obs)
        #action = np.array(action, dtype=int)
        #action = action.squeeze()
        next_obs, reward, done, _ = env.step(np.asarray(action).reshape(1,))

        #obs, reward, done, _ = env.step(torch.from_numpy(np.asarray(action)))

        R += reward
        t += 1
        reset = t == max_episode_len
        #print(type(agent))
        #agent = np.array(agent)
        #agent = torch.from_numpy(agent)
   
        agent.observe(
         next_obs,
         reward, 
         done, 
         reset
         )
        if done or reset:
            break
    if i % 10 == 0:
        print('episode:', i, 'R:', R)
    if i % 50 == 0:
        print('statistics:', agent.get_statistics())
print('Finished.')</code></pre>
Function report error for image size over [110000,80000]

username: yajun_li
cooked: <p>when i use F.affine_grid to tranform image size which’s size over [110000,80000],it<br>
report an error, intel MKL ERROR: Parameter 4 was incorrect on entry to cblas_sgemm.</p>

username: eqy
cooked: <p>This seems like it may be caused by some shapes/indexing being done in 32 bit values when a dimension is larger than that; could you open an issue here: <a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">Issues · pytorch/pytorch (github.com)</a> to track this issue?</p>
Embedding after Gumbel-Softmax

username: barthelemymp
cooked: <p>Hello,</p>
<p>I am trying on a model while during training one of the step is to sample some sequence and I need to be able to backpropagate through this step.<br>
To do so I am sampling using <code>F.gumbel_softmax(logits, tau=1, hard=True, dim=2)</code><br>
My problem is that I need to evaluate some score on this sampled sequences, and to do so I need to plug them back inside the model, which start by an Embedding layer (which needs the input to be in the format of indices not onehot) and doing an argmax to get the indices of the<code> 1</code>, will break the backprop.</p>
<p>Clearly it is computable, but I don t really know how to do that nicely… I was thinking of taking the weights of the <code>nn.Embedding</code> and do some matrix multiplication…</p>
<p>I would be thanksfull for any advice.</p>
<p>b</p>

username: mnoukhov
cooked: <p>If your embedding is standard then you can use <code>torch.matmul</code> as seen here <a href="https://github.com/facebookresearch/EGG/blob/170e5fe63c13244121a5b29b9bfb4870a0f11796/egg/core/gs_wrappers.py#L203" rel="noopener nofollow ugc">https://github.com/facebookresearch/EGG/blob/170e5fe63c13244121a5b29b9bfb4870a0f11796/egg/core/gs_wrappers.py#L203</a></p>
<p>Otherwise, you have to also implement the Embedding. E.g. if your <code>padding_idx != 0</code>, I’m not sure but I think you have to figure out how to make it not contribute to the gradient</p>
Is there a way to get all Pytorch Python API symbols?

username: Zysora
cooked: <p>Hello, a new user of Pytorch here, is there a place that contains all PyTorch symbols similarly to Tensorflow? Example: <a href="https://www.tensorflow.org/api_docs/python/tf/all_symbols" rel="noopener nofollow ugc">https://www.tensorflow.org/api_docs/python/tf/all_symbols</a></p>
<p>I know that the documentation already has the symbols and everything in multiple pages. I am looking for a way to gather all of them quickly so is there a page with all of it?</p>

username: huahuanZ
cooked: <p>If what you want is only the list without links. There is a hack way.</p>
<pre><code class="lang-auto">import torch
import importlib
import sys


def traversedir(module, pref=''):
    for attr in dir(module):
        if attr[0] == '_':
            continue
        if attr not in sys.modules:
            fullname = f'{pref}.{attr}'
            yield fullname
            try:
                importlib.import_module(fullname)
                for submodule in traversedir(getattr(module, attr), pref=fullname):
                    yield submodule
            except:
                continue


if __name__ == "__main__":
    for x in traversedir(torch, pref='torch'):
        print(x)

</code></pre>

username: Zysora
cooked: <p>Thank you so much <a class="mention" href="/u/huahuanZ">@huahuanZ</a> ! I would like to ask is it possible to only get public functions usable from a client? This returns every possible function in PyTorch.</p>

username: huahuanZ
cooked: <p>What do you mean “public functions” ?</p>
<p>I have filtered out those module/attribute names start with underscore.</p>

username: googlebot
cooked: <p>Older versions had single page docs like <a href="https://pytorch.org/docs/1.6.0/torch.html" class="inline-onebox" rel="noopener nofollow ugc">torch — PyTorch 1.6.0 documentation</a> ; three main pages are “torch”, “Tensor” (object methods) and “torch.nn”, most APIs are still the same.</p>
<p>And there is also “import torch; help(torch)” way</p>

username: Zysora
cooked: <p>What I mean by public functions are the ones invokable from a client like what the last post said. torch.nn and etc</p>

username: cdzhan
cooked: <p>May this link <a href="https://pytorch.org/docs/stable/genindex.html" class="inline-onebox" rel="noopener nofollow ugc">Index — PyTorch 1.10 documentation</a> be helpful</p>
Error loading 'caffe2_detectron_ops.dll' in windows

username: kranthi_akssy
cooked: <p>Hi, I’m new to PyTorch. I have installed using the command ‘conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch’ from <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a>. I’m using Jupyter IDE.<br>
When I try to import torch, the follwoing error is coming.</p>
<p>“OSError: [WinError 182]  Error loading “C:\ProgramData\Anaconda3\lib\site-packages\torch\lib\caffe2_detectron_ops.dll” or one of its dependencies.”</p>
<p>Can somebody help me to resolve this?</p>
<p>Thanks in advance.</p>

username: Pawan_Valluri
cooked: <aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/35803#issuecomment-725285085" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/35803#issuecomment-725285085" target="_blank" rel="noopener nofollow ugc">FileNotFoundError - caffe2_detectron_ops.dll on Windows source build if Python 3.8 used</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-01" data-time="13:37:28" data-timezone="UTC">01:37PM - 01 Apr 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-04-02" data-time="11:22:26" data-timezone="UTC">11:22AM - 02 Apr 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/realiti4" target="_blank" rel="noopener nofollow ugc">
          <img alt="realiti4" src="https://avatars.githubusercontent.com/u/43709619?v=4" class="onebox-avatar-inline" width="20" height="20">
          realiti4
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Hi, I'm getting this dll import errors when I'm trying to import torch since 1.4 if I use Python 3.8 for...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This answer worked for me.</p>
<p>Just deleting “<strong>caffe2_detectron_ops.dl</strong>l” from the path (“C:\Users\Girish\AppData\Local\Programs\Python\Python38\lib\sitepackages\torch\lib\caffe2_detectron_ops.dll”)</p>

username: shon
cooked: <p>Thank you for your suggestion. I also suffer from such a problem and fixed it as you suggested.</p>
Setting MLP for regression of 5 dimensional data

username: Bloopit
cooked: <p>I have a set of data that consists of 280k datapoints and each datapoint has 5 parameters (latitude, longitude, unixtime, wind speed and wind bearing). I use regression to fit a model to this dataset so I can output the wind speed and wind bearing at locations between locations in the original dataset.</p>
<p>I have done this to a degree, but upon testing I have found the outputs are pretty far off from expected.<br>
For example: From the dataset at coords: -37.7276, 144.9066, 1483174800, (lat, long, unix time). I should get windspeed = 16.6 and bearing = 193. But instead when I put the coords into the trained model I get [189.1, 310.24]. I assume windspeed = 189 and bearing = 310 because this is how i setup my y-data for input.</p>
<p>Any help would be greatly appreciated</p>
Determine inter-layer channel dependency

username: ralphmao
cooked: <p>I want to implement a function that modifies the output channel of a layer on the fly. To ensure that the model is still valid after the modification, I will need to find out all layers that depends on that layer (and there can be trivial layers/functions like relu between them). Is there any existing infra/function in PyTorch that can help me with this functionality?</p>
Is it possible to add gradient of an image to a patch

username: binbbaz
cooked: <p>I have an image gradient with size (224, 224) and a patch of (1, 768).</p>
<p>Both of them are not of the same size. Is it possible to add the image gradient to the patch and get an output of the patch i.e (1, 768)?</p>

username: ptrblck
cooked: <p>It would be possible to add both tensors after you’ve matched the size of them.<br>
E.g. you could reduce the image gradient to get 768 values before applying the elementwise addition.</p>

username: binbbaz
cooked: <p>Yes. Thank you, I’ve done that already. I just want to know if it’ll be possible to embed the whole gradient of the image in just one patch.</p>

username: ptrblck
cooked: <p>I don’t understand the question completely. Could you post an example how the gradient would be “embedded” into the patch?</p>

username: binbbaz
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a1f5e1093d63164dbb9c23fa88495956910b08d1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a1f5e1093d63164dbb9c23fa88495956910b08d1" title="img"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1f5e1093d63164dbb9c23fa88495956910b08d1_2_690x479.jpeg" alt="img" data-base62-sha1="n6LFiLjk5eRFsueO21YziwWVIpX" width="690" height="479" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1f5e1093d63164dbb9c23fa88495956910b08d1_2_690x479.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1f5e1093d63164dbb9c23fa88495956910b08d1_2_1035x718.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/1/a1f5e1093d63164dbb9c23fa88495956910b08d1_2_1380x958.jpeg 2x" data-dominant-color="9F9A90"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">img</span><span class="informations">1920×1334 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the end product would be the same patch on the left with the gradient of an entire image on the right added to it. My understanding is that it’s not possible, but knowledge isn’t bounded.</p>

username: binbbaz
cooked: <p>So, I figured out to do it myself. I divided the image gradient  (right) into 16 x 16 patches, created a loop that adds each patch to the original image patch (left). This way, I was able to add a 224 x 224 image gradient into a 16 x 16 patch. I just wanted to check out something. That was why I tried it</p>
How to recreate the below lines written in tensorflow to pytorch

username: Sai_Kishore
cooked: <pre><code class="lang-auto">global_step = tf.train.get_or_create_global_step()
    dec_learning_rate = tf.compat.v1.train.exponential_decay(flags.learning_rate, global_step,50000, 0.5, staircase=True)
    optim = tf.train.AdamOptimizer(learning_rate=dec_learning_rate)

    gvs = optim.compute_gradients(loss)
    gvs = [(tf.clip_by_norm(grad,1), val) for grad,val in gvs]
    optim = optim.apply_gradients(gvs)
</code></pre>

username: anantguptadbl
cooked: <pre><code class="lang-auto">def my_hook(grad):
    grad = grad.clone()
    grad = torch.clamp(grad, min=0, max=1)
    return grad

for cur_param in model.parameters():
    cur_param.register_hook(my_hook)
    
loss = torch.nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
my_lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=optimizer, gamma=0.99)
    
for cur_epoch in range(10):
    optimizer.zero_grad()
    output = model(X)
    final_loss = loss(output, y)
    final_loss.backward()
    optimizer.step()
    my_lr_scheduler.step()
    print("Epoch {0} Loss is {1}".format(cur_epoch, final_loss.item()))
</code></pre>
Why is requires_grad here false

username: Luca_Pamparana
cooked: <p>I have a NN  module which has the following parameters:</p>
<pre><code class="lang-auto">def __init__():
        self.a = torch.nn.Parameter(torch.randn(1))
        self.b = torch.nn.Parameter(torch.randn(10))
        self.c = torch.nn.Parameter(torch.randn((10, 1)))
        self.d = torch.nn.Parameter(torch.randn(10))
</code></pre>
<p>Now in my forward function I do something like:</p>
<pre><code class="lang-auto">def forward(z):
   out = list()
   for i in z:
       yh.append(self.a + torch.sum(self.b * torch.tanh(self.c @ i + self.)))

   return torch.Tensor(yh).view(z.shape[0], 1)
</code></pre>
<p>Now, this output tensor has gradients turned off. Is there some specific thing for converting a list of tensors to tensor with gradients?</p>

username: thecho7
cooked: <p><code>torch.Tensor</code> does not have a gradient in default.<br>
You are converting a gradient-tensor to non-gradient tensor by casting <code>torch.Tensor</code><br>
I recommend you to use <code>torch.cat</code> or some other functions instead of <code>torch.Tensor</code>.</p>