Question:
I’m a developer of a commercial application that requires initializing CUDA between dozens of independent processes. These processes are containerized for versioning - I can’t share much more, but using a single runtime to load several models isn’t possible.
However, initializing CUDA uses upwards of 2GB of RAM (not GPU memory). This is testable like so:
import torch
a = torch.tensor([0], device="cuda")

and monitoring RAM from any system tool. From my understanding, this is essentially all driver and library code, and thus the C++ runtime wouldn’t be significantly better.
I’ve tried ONNX (onnxruntime-gpu) and TensorRT in Python. They use about 1.5GB and 1.1GB of RAM respectively, which is still too much for my application. As people are deploying models on mobile devices I’m assuming there must be inference engines that are less memory intensive, but I haven’t found any in my searching that are compatible with NVIDIA GPUs.
Anyone run into similar problems or have any advice? How are people typically deploying models?

Context:
This is really a CUDA issue, see thediscussion in

      github.com/pytorch/pytorch

      Massive initial memory overhead GPU 97

        opened

        Oct 19, 2018

          closed

        Apr 15, 2019

          davidmascharka

  🐛 Bug
There is a huge RAM overhead for using the GPU even for processing small tensors.
Here's a standalone script:
# test.py
import torch
import...

One option might be to remove unneeded kernels. This is done for mobile already and discussed here:

      github.com/pytorch/pytorch

      [Feature Request] reduce CUDA runtime size by selectively compiling PyTorch GPU kernels 78

        opened

        Jan 6, 2020

          xxradon

  🚀 Feature
pytorch massive initial memory overhead GPU,we need a way to control the memory and gpu memory during pytorch to cuda()...

    module: binaries
    module: build
    module: cuda
    triaged

It’s not too hard to leverage the mobile work if you know how to do it.


Question:
Hi, I have had a few failures, trials and errors with conda and the compile and just wondered if there are any completed wheels for Pi-OS-64
Its available even if Beta in desktop & lite form https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2020-08-24/ 16
Anyone got any links, tutorials or even an official method as that would be so sweet.
I have failed on debian Arm64 & Ubuntu following some gists I found and the length of time it takes is killing my resolve.
Are there any binaries anywhere or would anyone post me to a recent and known working install for raspios_arm64
Thanks all

Context: Heya, I have a rather old PyTorch build for Debian 64 bit (http://mathinf.com/pytorch/arm64/ 10), but I’ve ordered a 8GB Raspberry Pi to ease some of the pain of building it and will see to having PiOs-64 builds.


Using PyCharm to debug Pytorch model on GCE, AWS or Azure

Question:
Hello everyone,
I know my question is not directly related to deep learning or Pytorch, but I have found this community so helpful and one time more I need your help.
I have implemented a model using Pytorch, but I do not have access to GPU on my local environment and using Google Colab is so frustrating because of lack of access to debugging utilities.
Now I come across this idea to use Google Preemptible Compute Engine or Amazon EC2 Spot or other cloud services, to test my implementation.
The problem is I am not sure about connecting cloud runtime to my local IDE and run debugs by using breakpoints in my local IDE.
I have read about this in the docs on clouds and PyCharm and it seems it is possible, but I am not sure about it.
Thanks for your replies.

Context:
Nikronic:

But, are you sure about Google colab?

Yes, I’m sure. Done that several times. Having free K80 GPU right in PyCharm is super handy when you want to make your code GPU proofed.
Configuration is tricky, multistep process, but doable. The main problem is that Colab VM instance is hidden behind a firewall. But inside of the virtual machine lives Linux Ubuntu, co we can do some hocus-pocus.
High level approach would be to:

Create “setup ssh forwarding” jupyter notebook in Google Colab. You can use my configuration notebook as a template for yours - demo_colab_ssh_access 56. Make sure you generate and then use your RSA access keys, customize port numbers, etc.
Setup and start ssh server on running Colab VM instance.
Establish ssh tunnel from Colab VM instance to some server on the internet, forward traffic from port 22 to some port there and…
Expose this port for external access. portmap.io 15 service allows one port forwarding for free.
From now on you have ability to ssh from your computer to Colab VM instance and you can proceed with configuring remote interpreter via ssh in PyCharm 36.
Every time you start new Colab session you’ll just replay “setup ssh forwarding” jupyter notebook and voila

Ping me if you get stuck.


Simple producer-consumer problem causing 100% CPU usage

Question:
The below code is causing 100% CPU usage.
There’s no GPU.
import random
from queue import Queue
from threading import Thread

import torch
from torchvision.models import resnet18
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity

model = resnet18(pretrained=True)
model.fc = torch.nn.Identity()

#list of 50 image tensors of varying sizes
ims = [torch.randn(1,3,random.randint(224,512),random.randint(224,512)) for i in range(50)]

def producer(out_q):

    for im_tensor in ims:
        with torch.no_grad():
            feat_vector = model(im_tensor).detach().numpy()
        out_q.put(feat_vector)

def consumer(in_q):
    feat_vector1 = in_q.get()
    feat_vector2 = np.random.randn(1,512)
    sim_score = cosine_similarity(feat_vector1,feat_vector2)[0][0]
    in_q.task_done()
    return sim_score

for i in range(50):
    q = Queue()
    t1 = Thread(target = consumer, args =(q, ))
    t2 = Thread(target = producer, args =(q, ))
    t1.start()
    t2.start()
    q.join()

How to avoid the CPU cranking up?

Context:
You are not using the GPU, so a high CPU workload is expected.
This tutorial 6 explains how to use the GPU by pushing the model as well as the tensors to the device.


Torchserve stopped. Failed to bind. Address already in use

Question:
Hi,
I am trying to get Torchserve running locally on my Ubuntu Machine. But I encounter this error after loading the model:
2021-03-18 18:31:38,285 [INFO ] main org.pytorch.serve.ModelServer - Torchserve stopped.
java.io.IOException: Failed to bind
        at io.grpc.netty.shaded.io.grpc.netty.NettyServer.start(NettyServer.java:264)
        at io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
        at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
        at org.pytorch.serve.ModelServer.startGRPCServer(ModelServer.java:396)
        at org.pytorch.serve.ModelServer.startGRPCServers(ModelServer.java:377)
        at org.pytorch.serve.ModelServer.startAndWait(ModelServer.java:116)
        at org.pytorch.serve.ModelServer.main(ModelServer.java:95)
Caused by: io.grpc.netty.shaded.io.netty.channel.unix.Errors$NativeIoException: bind(..) failed: Address already in use
Exception in thread "Thread-0" java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
        at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
        at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
        at io.netty.channel.AbstractChannel.close(AbstractChannel.java:232)
        at org.pytorch.serve.ModelServer.stop(ModelServer.java:473)
        at org.pytorch.serve.ModelServer$1.run(ModelServer.java:91)

I dont have anything running on 9000, 9001, 8081, 8082 or any other port.
Following is my config:
Torchserve version: 0.3.0
TS Home: /home/pc/anaconda3/envs/torch/lib/python3.7/site-packages
Current directory: /home/pc/lezwon
Temp directory: /tmp
Number of GPUs: 2
Number of CPUs: 20
Max heap size: 16008 M
Python executable: /home/pc/anaconda3/envs/torch/bin/python
Config file: config.properties
Inference address: http://127.0.0.1:8080
Management address: http://127.0.0.1:8081
Metrics address: http://127.0.0.1:8082
Model Store: /home/pc/lezwon/model_store
Initial Models: car_roi=car_roi.mar
Log dir: /home/pc/lezwon/logs
Metrics dir: /home/pc/lezwon/logs
Netty threads: 0
Netty client threads: 0
Default workers per model: 2
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: false
Metrics report format: prometheus
Enable metrics API: true

Torchserve runs fine through a docker container. Only fails locally. Any help here would be appreciated. Thanks.

Context: Fixed it. There was AnyDesk running on port 7070. Closed it and it now works fine.


Time-series prediction day-by-day with Conv1D

Question:
Hello
I developed a standard Conv1D model in Pytorch to predict time series with classification (4 classes).
I gathered a train set (5000 data) and a test set (1000 data). The model predicts daily data by batches and is quite efficient.
As the results were satisfactory, I then moved to the next step :

I trained my model
I saved the model
I used the trained model on daily new data (bringing data for prediction day-by-day, instead of by batch size).
The results were very deceptive (if not catastrophic).

Therefore, I checked what happen and tried to save my trained model, to clean the GPU cache, to re-load the model and to apply it to the test-set.
Herewith the original code :
    train_dl, test_dl = get_data_loaderRN(X_train, y_train, X_tests, y_tests, batch_size)
    model = Conv_1D(input_shape, nb_classes, num_features, seq_length, batch_size, iter_model, iter_pre, dropout)
    model = model.double()
    model.cuda()
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr= learning)
    y_predtrain, running_loss = train_Conv_1D(model, criterion, optimizer, epochs, learning, verbose, train_dl)
    acc_train = accuracy_score(y_train, y_predtrain)
    y_predsubt, y_pred_proba, running_loss = eval_model_a(model, nb_classes, criterion, test_dl)
    acc_test = accuracy_score(y_tests, y_predsubt)

And the code with save and re-load model :
    train_dl, test_dl = get_data_loaderRN(X_train, y_train, X_tests, y_tests, batch_size)
    model = Conv_1D(input_shape, nb_classes, num_features, seq_length, batch_size, iter_model, iter_pre, dropout)
    model = model.double()
    model.cuda()
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters(), lr= learning)
    y_predtrain, running_loss = train_Conv_1D(model, criterion, optimizer, epochs, learning, verbose, train_dl)
    acc_train = accuracy_score(y_train, y_predtrain)
    ## Save model and clean GPU cache + re-seed
    torch.save(model, model_path)
    torch.cuda.empty_cache()
    torch.manual_seed(my_seed)
    torch.cuda.manual_seed(my_seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    ## Reload model and use it to predict
    model = torch.load(model_path)
    model.to(device)
    y_predsubt, y_pred_proba, running_loss = eval_model_a(model, nb_classes, criterion, test_dl)
    acc_test = accuracy_score(y_tests, y_predsubt)

The Save / Clean / Load steps are there to simulate what happened with the model used for daily prediction… and the results are far from those with the original model.
How could I address this problem ?
Thanks in advance for your support.
Best,
NB1. The output is double “y_predsubt” recors the predicted class and “y_pred_proba” records the likelihood associated by the model to each of the 4 classes.
NB2. I did the same with a LSTM model and get the same issue

Context:
If I understand the issue correctly, you are seeing a performance drop after loading a saved model and evaluating it on the same dataset?
If so, are you calling model.eval() while the evaluation is performed? If that’s the case, could you pass a static input to the model before saving and after loading (e.g. torch.ones) and compare the outputs?


Error deploying torchserve: model load failed

Question:
Here is the dropbox link to the zip containing log file, model.py, handler.py etc.
https://www.dropbox.com/s/gi8wmyxmpy1zx84/Files.zip?dl=0 1
Model trained:

image794×230 8.89 KB

Commands used:
torch-model-archiver --model-name foodnet_resnet18 \
                     --version 1.0 \
                     --model-file model/model.py \
                     --serialized-file model/foodnet_resnet18.pth \
                     --handler model/handler.py \
                     --extra-files model/index_to_name.json

torchserve --start \
           --ncs \
           --ts-config deployment/config.properties \
           --model-store deployment/model-store \
           --models foodnet=foodnet_resnet18.mar

Context: The loading behavior is not defined by torchserve but by PyTorch depending how you’ve stored the state_dict. You could either push the model to the CPU before storing the state_dict or use the map_location argument as explained here 10.


Question:
I’m working on an object detection model to be deployed on Raspberry pi.
I usually stick to tflite for this use case. But I really want to use PyTorch if possible.
I’ll be running inference on Raspberry pi. Currently, what is the best way to go about doing this?
Will torchscript help?
Most discussions in the forums give an arm_whl file to setup PyTorch in raspberry pi, is this the best method  - Installing pytorch on raspberry pi 3 89?
Any resource on this topic will be very much appreciated. Thanks.

Context:
Haven’t used tflite, but I know since pytorch 1.3 you can use pytorch on mobile devices / arm architectures for experimental usage.
In Pytorch 1.5 you can use dynamic quanization for better latency.
Here 152 is the tutorial.
Also huge advantage of detr object detector is the model size. Check it out here 75.


Export libtorch model which support variable input tensor

Question:
Do libtorch model support variable size of the tensor? Trying to train a classification on medical images(CT scans), but the number of slices are different for each file([512, 512, variable]). Is it possible to export the model witch support input of [1, variable, 512, 512]?
Every examples of libtorch use fix size input before export the model
img = torch.zeros((1,3,16,112,112)) #64,3,16,112,112; 16, 3, 1, 112, 112
traced_script_module.save("video_r3d_18.pt")

Thanks

Context: Yes, it should be possible via torch.jit.script. Tracing via torch.jit.trace uses the passed input to trace all operations and might fail, if you are using different inputs in the deployment use case.


Object detection model deployment on CPU? Real-life solutions

Question:
Hi,
I am currently working on a small toy-project that involves object detection as one of the steps.
Currently, I amusing a pre-trained Faster-RCNN from Detectron2 10 with ResNet-101 backbone.
I wanted to make an MVP and show it to my colleagues, so I thought of deploying my model on a CPU machine. Detectron2 can be easily converted to Caffe2 (DOCS 20) for the deployment.
I measured the inference times for GPU and the CPU mode. The inference time of the original Detectron2 model using PyTorch and GPU is around 90ms on my RTX2080 Ti.
The converted model on CPU (i9 9940X) and using Caffe2 API took 2.4s. I read that the Caffe2 is optimized for CPU inference, so I am quite surprised by the inference time on CPU. I asked about this situation on the Detectron2 GitHub and I got an answer like: „Expected inference time of R-50-FPN Faster R-CNN on a 8 core CPU is around 1.9s. Usually, ResNets are not used on CPUs.”
There is my question, how such deep learning solutions for Computer Vision should be deployed in the real-world? I read somewhere that Facebook does use Caffe2 for their production models as CPUs are super cheap compared to GPUs (of course they are), but the difference in the running time is really huge. Using CPU for object detection seems useless for any real-time application.
Should I use some other architecture, which does not include ResNet or Faster-RCNN (like YOLO v4/v3, SSD, etc.)? Or maybe the original GPU-trained model should be converted to ONNX and then used in other more CPU-optimized frameworks such as OpenVINO? Or there are some other tweaks such as quantization, pruning, etc. that are necessary to boost the CPU-inference efficiency of production models?
I know that this is just a toy-project (for now at least), I can use GPU for inference (quite costly in real applications?) or just use other architecture (but sacrifice performance). I am just wondering what is the go-to solution for real-world systems.
Thanks in advance for sharing your knowledge and experience. I will be grateful for any hints!

Context:
I am not a big expert in this area but just for letting know what I have discovered.
Converting  PyTorch models to TorchScripts models is a good idea. You get a file that contains the arch class and weights. In the first run of the model gets optimized and after that is pretty fast. I am getting around 1.98 s ± 42.1 ms in my Ryzen 1700 with Mask-RCNN Resnet 50 (the implementation of torchvision).
I have also discovered quantization 31 . It optimizes models and make them able to run on a mobile or arm devices like Raspberry. However, after quantization the model losses some precision. Here you have an image explaining the process:
image1396×1574 113 KB
You could run this models on your computer not just mobile devices


Question:
We use bazel 3 as part of large monorepo to integrate with torch.  In order to support large amounts of concurrent builds, we must execute our build actions remotely and this entails serializing the files needed for the build action.  However, protobuf serialization has a hard limit of 2gb which makes the libtorch_cuda.so unable to be used remotely, which in turn makes many actions in our repository unable to execute remotely.  Running these actions locally would be very bad for build performance, so I was looking for ways to shrink libtorch_cuda.so.
I tried stripping out debug symbols:
strip --strip-debug libtorch_cuda.so
strip --strip-unneeded libtorch_cuda.so
no effect.
Do you have any suggestions?

Context:
Solved by ptrblck in post #21

                The additional compute capabilities might be displayed, since I assume you are linking the CUDA runtime as well as cudnn statically into the wheel. Without removing these architectures in e.g. libcudnn.a via nvprune, these archs should be displayed in the wheel.
Note that I posted the command to bu…


Fail to “import torch” on Windows debug build

Question:
I try to build pytorch from source on Windows, using Anaconda3, Ninja, and “DEBUG=1”. “python setup.py install”. It failed with fatal error LNK1104 “LNK1104: ╬▐╖¿┤≥┐¬╬─╝■í░python37_d.libí▒”. I found the  “python37_d.lib” is not in Anacoda. So in order to get “python37_d.lib” available, I installed python3.7 from www.python.org 1 and select “Download debug binaries”. Then I copied the whole newly installed python folder to Anaconda3\envs\gt37 and replace existing old files.
After seting enviroment variable “LIB” to the Anaconda3\envs\gt37\libs, I successully built the debug version.
However, when I execute “import torch” in python, it shows the following error:

(gt37) C:\Users\tegao\mygit\v0512\pytorch>python
Python 3.7.6 (tags/v3.7.6:43364a7ae0, Dec 19 2019, 00:42:30) [MSC v.1916 64 bit (AMD64)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>>\ import torch
Traceback (most recent call last):
File “”, line 1, in
File “C:\Users\tegao\mygit\v0512\pytorch\torch_init_.py”, line 247, in
from .random import set_rng_state, get_rng_state, manual_seed, initial_seed, seed
File “C:\Users\tegao\mygit\v0512\pytorch\torch\random.py”, line 4, in
from torch._C import default_generator
ImportError: cannot import name ‘default_generator’ from ‘torch._C’ (unknown location)

Then I tried using python3.8, still failed with same error.
Then I tried using “python setup.py develop” and set the following enviroments：
set DEBUG=1
set USE_DISTRIBUTED=0
set USE_MKLDNN=0
set USE_CUDA=0
set BUILD_TEST=0
set USE_FBGEMM=0
set USE_NNPACK=0
set USE_QNNPACK=0
set USE_XNNPACK=0
set LIB=C:\Users\tegao\Anaconda3\envs\gt38\libs
This time the error changed when “import torch”:

(gt38) C:\Users\tegao\mygit\pytorch>python
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:37:02) [MSC v.1924 64 bit (AMD64)] on win32
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import torch
Fatal Python error: _PyInterpreterState_Get(): no current thread state
Python runtime state: unknown

I make sure I have run “python setup.py clean” before each try.
But it could succeed on Ununtu. Building release version of pytorch on Windows could also succeed.
Is my method wrong(using self-installed python files to replace the ones under Anaconda in order to have pytorch37_d.lib)? How to successfully build debug version of pytorch on windows?

Context:
I had this issue before.
I uninstalled the anaconda
I installed again. But this time i selected the pip and specifically if you don’t have cuda select NONE
Screen Shot 2020-05-15 at 1.56.33 PM1862×1332 216 KB


Unifying Preprocessing Pipelines for Machine Learning on Time Series Data (High Throughput Batch & Low Latency Streaming)

Question:
The Question/TL;DR
What frameworks, design patterns, systems, etc. exist for unifying the preprocessing of time series data in Python such that high throughput is achieved on the retrospective batch data for training machine learning models while also allowing for easy model deployment with low latency on real time streaming data for inference?
System Details
The data is raw, high frequency (50-500Hz) ICU bedside monitor signals which are recorded into a bespoke time-series database containing over 3 trillion data points which come from a RabbitMQ stream producing on the order of tens of thousands of new points per second. The database is accessible via a rather slow REST API backed in PHP or SDKs based in various languages. Researchers are really only familiar with Python based tooling (numpy, scipy, pandas, biosppy, PyTorch, Tensorflow, various GitHub repos of domain specific code from research papers, etc) while also not being interested or skilled in engineering custom deployment code for each of their models. We are needing high throughput while preparing large subsets (up to a couple terabytes raw) of the database for efficiently and easily developing models from retrospective data, while also having a very low latency (<500ms) on real time streaming data.
Options

Traditional ML Research Approach

Relevant retrospective data is pulled to disk, processed sequentially (usually inefficiently), and eventually a decent performing model is made.
This works, but is completely incompatible with the real time systems meaning deployment requires significant custom engineering. Or worse, this processing is simply inefficient and takes days to process the dataset and/or extra engineering work to parallelize. Therefore, this is not a great approach due to how the research phase is slowed down and the excessive technical work requirements for optimizing the processing as well as deploying to real time data.

Custom Python Framework

I had started on developing a Python based framework which tried to minimize the amount of systems knowledge and code required from researchers while still allowing them freedom to perform their work however they saw fit via familiar tools (ie, Python). This ended up being quite the undertaking and has been very difficult. Worries of bugs, dealing with edge cases, performance, maintenance, adaptability, … have me looking for a better approach. Many of the problems I discovered along the way seem to be addressed by the streaming frameworks.

Streaming Frameworks

It appears that streaming frameworks are the current leading way of feeding real time machine learning models. In particular, Spark Streaming, but also Storm, Flink, Samza, Trill, …

Apache Beam 9 is currently the most interesting though I am uncertain this is the best option

Unified: batch and streaming can be processed in the same way

I recently learned Flink offers similar functionality

Extensible: allows for new SDKs, IO connectors, etc

This is important for interfacing with our bespoke database.
Issue: The Python SDK of Apache Beam does not have existing IO for RabbitMQ but Java SDK does
Allows for the execution of any Python code on the data, not just simplistic min/max/windowing and such of other streaming engines

Language: Allows for everything to be written in Python

Only other all Python offering I have found is Streamz 7

Thank you for your time. If there are better venues or ways to ask this question, please let me know! Especially if I have misused terminology, please let me know as that will allow me to research these issues better on my own.

Context:
i’m curious what you ended up with.
Currently looking at Flink as a means of upgrading our proprietary framework built on python DAG graph processing. This was developed because Python is the data science standard in our company and re-engineering of the features built in research stages would be very inefficient as well as prone to errors due to mismatches in processing algorithms in the different languages.
Using the same processor for batch and streaming will go a long way, although the python implementation of Flink (v1.12) does require a bit of research to use because the documentation and examples on the internet can be a bit sparse (Also some bugs…).
We’re moving towards Azure because of the IOT edge capability maturity, but running Flink  there would require a lot of setup and management of quite low level infra stuff.
Anyway, i’d be interested in hearing what you decided.


Is there any pytorch resources for Click Though Rate?

Question: It seems that the resources about CTR(Click-Though Rate) is rare.

Context:
Hi,
I’m not sure what you mean by this? You mean a Dataset for training?


How to use pre-trained model built using Pytorch?

Question:
I am using a simple perceptron based classifier to generate sentiment analysis in Pytorch, complete code here Classifying Yelp Reviews 1.
The example does sentiment analysis, it outputs if the given input string is positive or negative.

example: this is a pretty small old great book -> positive

The application stores the final model along with the vectorizer.json.
So my question is: What should be the pre-requisites to build a separate application just to test the model, so that it can be used in web-application later-on?
Below is my current understanding and queries for this:
I assume to test, we need to load the model, load model parameters and evaluate for inference, please confirm

Context: I think this tutorial 9 is pretty straightforward in explaining how to load a pre-trained model for inference. You would need the ‘state-dict’ stored in the disc.


Question:
Hi,
I am using a pretty bodged method of saving images from a 3rd party application, calling my Python script for inference referencing those images, and then importing the result into the 3rd party application again. These calls are made through Javascript with a script that only lives for a few milliseconds. So each time I call the python script it needs to load my model into memory again.
Is there any way to keep a model in memory between my calls to inference script?
Maybe it could be possible to call one script that loads the model into memory and another one that performs inference (and can somehow access that model)?

Context: The best option seems to be to create a local webserver.


PyTorch to ONNX export - ONNX Runtime inference output (Python) differs from PyTorch

Question:
Hi there, I tried to export a small pretrained (fashion MNIST) model to ONNX for test cases and evaluated the results. The outputs were completely differnt and I already tried different solutions which did not help to solve the problem.
I spent weeks on finding a solution to this error - may you please help me?
Here are the labels and a formatted output of the PyTorch model and the ONNX model which was run with ONNX Runtime
Labels: ['T-shirt', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', ' Bag', 'Ankle boot']

PyTorch output: [14.139054 -1.463651 3.0200443 5.6304793 -3.21046 -6.91372 10.828293 -12.109244 -0.4888268 -7.0172133]
ONNX Runtime Output: [5.685347 4.6847196 -4.7160497 -11.215544 -14.1343975 28.269392 1.3876779 -3.9011323 1.518218 0.49619618]

Below you can find the unformatted output and the used files.

Unformatted output
Label: T-shirt Confidence: 0.9645941 Time taken: 0.07566659999999992
PyTorch output: [ 14.139054   -1.463651    3.0200443   5.6304793  -3.21046    -6.91372
  10.828293  -12.109244   -0.4888268  -7.0172133]
ONNX Runtime output: [  5.685347     4.6847196   -4.7160497  -11.215544   -14.1343975
  28.269392     1.3876779   -3.9011323    1.518218     0.49619618]
========================================
Final top prediction is: Sandal with confidence of: 1.0
========================================
========================================
Inference time: 0.99 ms
========================================

Export routine
DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")
RAND_INPUT = torch.randn(BATCH_SIZE, 3, 112, 112, device=DEVICE)

torch.onnx.export(MODEL,                                            # model being run
                      RAND_INPUT,                                   # model input (or a tuple for multiple inputs)
                      str(MODEL.__class__.__name__)+".onnx",        # where to save the model (can be a file or file-like object)
                      export_params=True,                           # store the trained parameter weights inside the model file
                      #verbose=True,                                 # output graph to console
                      opset_version=11,                             # the ONNX version to export the model to
                      do_constant_folding=True,                     # whether to execute constant folding for optimization
                      enable_onnx_checker=True,
                      input_names = ['input'],                      # the model's input names
                      output_names = ['output'],                    # the model's output names
                      dynamic_axes={'input' : {0 : 'batch_size'},   # variable lenght axes
                                    'output' : {0 : 'batch_size'}})

Neural Network Model (mnist_model.py)
import torch
import torch.nn as nn
from torch.nn import init
import torch.nn.functional as F
import numpy as np

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return x

Testing routine (test.py)
import time
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
from torch.utils.data import Dataset, DataLoader
from PIL import Image

def bulkPredictImages(model: nn.Module, classes: list, dataLoader: DataLoader):
    resultArray = []
    resultConfidenceArray = []
    totalTime = 0.0
    for images, _ in dataLoader:
        if torch.cuda.is_available():
            images = images.cuda()
        for image in images:
            image = image.unsqueeze_(0)
            result, confidence, timeTaken = predictImage(model, classes, image)
            resultArray.append(result)
            resultConfidenceArray.append(confidence)
            totalTime += timeTaken

    return resultArray, resultConfidenceArray, totalTime

def predictImage(model: nn.Module, classes: list, inputImage: Image, transform = None):
    timeStart = time.perf_counter() #in seconds
    model.eval()
    with torch.no_grad():
        if transform != None:
            device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
            inputImage = transform(inputImage).float()
            inputImage = inputImage.unsqueeze_(0)
            inputImage = inputImage.to(device)
        output = model(inputImage)
        idx = output.data.cpu().numpy().argmax()
        label = classes[idx]
        top_p, top_classes = F.softmax(output, dim=1).topk(1, dim = 1)
        confidence = top_p.data.cpu().numpy().max()

        timeStop = time.perf_counter() #in seconds
        return label, confidence, timeStop - timeStart

def main():
    pass

if __name__ == '__main__':
    main()

Converting and evaluation (PyTorchToOnnxConverter.py) (please have mercy for my coding style)

import io
import os
import sys
import numpy as np

import time

from torch import nn
import torch.onnx

import torch.nn as nn
import torch.nn.init as init
import torch.nn.functional as F
from torchvision import transforms
from collections import OrderedDict

import onnx
import onnxruntime
from onnx import optimizer as optim

from PIL import Image

import test
import mnist_model

MODEL = None

BATCH_SIZE = 1
MODEL_PATH = ""
TEST_IMAGE_PATH = ""
LABELS = []

IMG = None

MONOCHROME = False
INPUT_LAYOUT = ()

DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

RAND_INPUT = None

TRANSFORM = None

def setMNIST():
    global MODEL
    global MODEL_PATH
    global TEST_IMAGE_PATH
    global LABELS
    global IMG
    global MONOCHROME
    global INPUT_LAYOUT
    global RAND_INPUT
    global TRANSFORM

    MODEL = mnist_model.Net()
    if torch.cuda.is_available():
        MODEL = MODEL.cuda()
    MODEL_PATH = "D:/Projekte/Masterthesis/DeepNerualNetworks/CustomFashionMNIST/best_model.pth"
    TEST_IMAGE_PATH = "D:/Projekte/Masterthesis/DeepNerualNetworks/PyTorchToOnnxConverter/tshirt.png"
    LABELS = ['T-shirt', 'Trouser', 'Pullover', 'Dress', 'Coat', 'Sandal', 'Shirt', 'Sneaker', ' Bag', 'Ankle boot']
    IMG = Image.open(TEST_IMAGE_PATH).convert('L')
    INPUT_LAYOUT = (1, 1, 28, 28)
    MONOCHROME = True
    RAND_INPUT = torch.randn(BATCH_SIZE, 1, 28, 28, device=DEVICE)
    TRANSFORM = transforms.Compose([
        transforms.Resize(size=28),
        transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))
    ])

def preprocess(input_data):
    # convert the input data into the float32 input
    img_data = input_data.astype('float32')

    #normalize
    mean_vec = np.array([0.485, 0.456, 0.406])
    stddev_vec = np.array([0.229, 0.224, 0.225])
    norm_img_data = np.zeros(img_data.shape).astype('float32')
    for i in range(img_data.shape[0]):
        norm_img_data[i,:,:] = (img_data[i,:,:]/255 - mean_vec[i]) / stddev_vec[i]

    #add batch channel
    norm_img_data = norm_img_data[np.newaxis, :, :, :]
    #norm_img_data = img_data[np.newaxis, :, :, :]
    norm_img_data = np.resize(norm_img_data, INPUT_LAYOUT).astype('float32')
    return norm_img_data

def softmax2(arr):
    sum = 0
    max = np.max(arr)
    print(max)
    for i in arr:
        print(i)
        sum += np.exp(i - max)    # subtract max in numr and denr to avoid overflow

    res = []
    for i in arr:
        res.append(np.exp(i - max) / sum)

    return res

def softmax3(inp):
    top_p, top_classes = F.softmax(inp, dim=1).topk(1, dim = 1)
    confidence = top_p.data.cpu().numpy().max()
    return top_p, confidence

def softmax(x):
    sum = 0
    print("ONNX Runtime output: " + str(x[0][0]))
    x = x.reshape(-1)
    e_x = np.exp(x - np.max(x))
    return e_x / e_x.sum(axis=0)

def postprocess(result):
    smRes = softmax(np.array(result)).tolist()
    return smRes

def onnxImageProcess(img, monochrome: bool = False):
    image_array = np.asarray(img)
    if monochrome:
        image_array = image_array[:, :, np.newaxis]
    image_array = image_array.transpose(2, 0, 1)

    return preprocess(image_array)

def runOnnxTest(ort_session, input_data):
    #image = Image.open(TEST_IMAGE_PATH)

    input_name = ort_session.get_inputs()[0].name

    start = time.time()
    raw_result = ort_session.run(None, {input_name: input_data})
    end = time.time()
    res = postprocess(raw_result)
    #res = softmax2(raw_result)
    #softmax3(raw_result)

    inference_time = np.round((end - start) * 1000, 2)
    idx = np.argmax(res)

    print('========================================')
    print('Final top prediction is: ' + LABELS[idx] + " with confidence of: " + str(res[idx]))
    print('========================================')

    print('========================================')
    print('Inference time: ' + str(inference_time) + " ms")
    print('========================================')

def to_numpy(tensor):
    return tensor.detach().cpu().numpy() if tensor.requires_grad else tensor.cpu().numpy()

def torchImageProcess(img, transform):
    inputImage = transform(img).float()
    inputImage = inputImage.unsqueeze_(0)
    return inputImage.to(DEVICE)

def compareOnnxModelWithTorch(session, onnxInput, torchOutput):
    ort_inputs = {session.get_inputs()[0].name: onnxInput}
    ort_outs = session.run(None, ort_inputs)

    # compare ONNX Runtime and PyTorch results
    np.testing.assert_allclose(to_numpy(torchOutput), ort_outs[0], rtol=1e-03, atol=1e-05)

def add_value_info_for_constants(model : onnx.ModelProto):
    """
    Currently onnx.shape_inference doesn't use the shape of initializers, so add
    that info explicitly as ValueInfoProtos.
    Mutates the model.
    Args:
        model: The ModelProto to update.
    """
    # All (top-level) constants will have ValueInfos before IRv4 as they are all inputs
    if model.ir_version < 4:
        return

    def add_const_value_infos_to_graph(graph : onnx.GraphProto):
        inputs = {i.name for i in graph.input}
        existing_info = {vi.name: vi for vi in graph.value_info}
        for init in graph.initializer:
            # Check it really is a constant, not an input
            if init.name in inputs:
                continue

            # The details we want to add
            elem_type = init.data_type
            shape = init.dims

            # Get existing or create new value info for this constant
            vi = existing_info.get(init.name)
            if vi is None:
                vi = graph.value_info.add()
                vi.name = init.name

            # Even though it would be weird, we will not overwrite info even if it doesn't match
            tt = vi.type.tensor_type
            if tt.elem_type == onnx.TensorProto.UNDEFINED:
                tt.elem_type = elem_type
            if not tt.HasField("shape"):
                # Ensure we set an empty list if the const is scalar (zero dims)
                tt.shape.dim.extend([])
                for dim in shape:
                    tt.shape.dim.add().dim_value = dim

        # Handle subgraphs
        for node in graph.node:
            for attr in node.attribute:
                # Ref attrs refer to other attrs, so we don't need to do anything
                if attr.ref_attr_name != "":
                    continue

                if attr.type == onnx.AttributeProto.GRAPH:
                    add_const_value_infos_to_graph(attr.g)
                if attr.type == onnx.AttributeProto.GRAPHS:
                    for g in attr.graphs:
                        add_const_value_infos_to_graph(g)

    return add_const_value_infos_to_graph(model.graph)

def remove_initializer_from_input(model):
    if model.ir_version < 4:
        print(
            'Model with ir_version below 4 requires to include initilizer in graph input'
        )
        return

    inputs = model.graph.input
    name_to_input = {}
    for input in inputs:
        name_to_input[input.name] = input

    for initializer in model.graph.initializer:
        if initializer.name in name_to_input:
            inputs.remove(name_to_input[initializer.name])

    return model

def main():
    setMNIST()
    state = 0
    so = onnxruntime.SessionOptions()
    so.graph_optimization_level = onnxruntime.GraphOptimizationLevel.ORT_ENABLE_ALL

    state = torch.load(MODEL_PATH, map_location=DEVICE)

    new_state_dict = OrderedDict()
    for k, v in state['state_dict'].items():
        head = k[:7]
        if head == 'module.':
            name = k[7:]  # remove `module.`
        else:
            name = k
        new_state_dict[name] = v

    MODEL.load_state_dict(new_state_dict)

    MODEL.eval()
    MODEL.is_training = False

    #torch.save(MODEL, "512.pt")

    result, confidence, timeTaken = test.predictImage(MODEL, LABELS, IMG, TRANSFORM)

    sys.stdout.write("Label: " + str(result) + " Confidence: " + str(confidence) + " Time taken: " + str(timeTaken) + "\n")

    #x = torch.randn(BATCH_SIZE, 3, 112, 112, requires_grad=True)

    # Export the model
    torch.onnx.export(MODEL,                                        # model being run
                      RAND_INPUT,                                   # model input (or a tuple for multiple inputs)
                      str(MODEL.__class__.__name__)+".onnx",        # where to save the model (can be a file or file-like object)
                      export_params=True,                           # store the trained parameter weights inside the model file
                      #verbose=True,                                 # output graph to console
                      opset_version=11,                             # the ONNX version to export the model to
                      do_constant_folding=True,                     # whether to execute constant folding for optimization
                      enable_onnx_checker=True,
                      input_names = ['input'],                      # the model's input names
                      output_names = ['output'],                    # the model's output names
                      dynamic_axes={'input' : {0 : 'batch_size'},   # variable lenght axes
                                    'output' : {0 : 'batch_size'}})

    onnx_model = onnx.load(str(MODEL.__class__.__name__)+".onnx")

    onnx_model = remove_initializer_from_input(onnx_model)

    #passes = ["extract_constant_to_initializer", "eliminate_unused_initializer"]
    #add_value_info_for_constants(onnx_model)
    #for init in onnx_model.graph.initializer:
    #    for value_info in onnx_model.graph.value_info:
    #        if init.name == value_info.name:
    #            onnx_model.graph.input.append(value_info)
    #onnx_model = optim.optimize(onnx_model, passes)

    onnx.save(onnx_model, str(MODEL.__class__.__name__)+".onnx")
    # Check that the IR is well formed
    onnx.checker.check_model(onnx_model)

    # Print a human readable representation of the graph
    graph = onnx.helper.printable_graph(onnx_model.graph)

    with open("graph_output.txt", mode="w") as fout:
        fout.write(graph)

    ort_session = onnxruntime.InferenceSession(str(MODEL.__class__.__name__)+".onnx", so)
    compareOnnxModelWithTorch(ort_session, to_numpy(RAND_INPUT), MODEL(RAND_INPUT))
    #compareOnnxModelWithTorch(ort_session, onnxImageProcess(IMG, MONOCHROME), MODEL(torchImageProcess(IMG, TRANSFORM)))

    res = to_numpy(MODEL(torchImageProcess(IMG, TRANSFORM)))
    print("PyTorch output: " + str(res[0]))
    runOnnxTest(ort_session, onnxImageProcess(IMG, MONOCHROME))

if __name__ == "__main__":
    main()

Thank you for your time and help

Context:
Could you test the PyTorch and ONNX model with a constant input, e.g. torch.ones, and check if the result still differs?
If not, I guess the preprocessing of the input data might be different, which would also change the model outputs.


Will pytorch support Apple’s new arm cpu?

Question: Tensorflow had decided to add the support of Apple Sillicon, will pytorch support too?

Context:
Probably yes
After all pytorch just as tensorflow has gained it’s fair share of popularity.


Keras gives better performance in both training speed and result generalization than pytorch in simple MLP, help!

Question:
Problem
I generated a small dataset, use simple MLP to fit this function in pytorch and keras to compare speed and performance. I define two networks with same setting(cpu, l2_reg, dropout, layers, weights initialization…)
The result is: keras has faster data traverse speed, less cpu usage and better generalization results.

Updated!
I set num_workers=4  of dataloader, each epoch time takes 1.44s, dataloader for loop takes 0.8s, the train time is comparable with keras. I also set torch.set_num_threads(8), the cpu usage is normal. But under the same setting, Keras still gets better results. There must be something wrong…
Screen Shot 2020-11-16 at 5.51.45 PM1476×478 45.5 KB
There must be something wrong.
I really enjoy the flexibility of pytorch,  can anybody help me where the problem is? thanks!
Train Code
pytorch
#####################generate data
#f(x) = x[:39:2]**3 * x[1:40:2]**2 + x[:40:4] + noise
import numpy as np
seed = 6
np.random.seed(seed)
train_x = np.random.random((300000, 80))
train_y = np.sum(train_x[:, :39:2]**3, axis=1) * np.sum(train_x[:, 1:40:2]**2, axis=1) + np.sum(train_x[:, :40:4], axis=1) #+ np.random.random(train_x.shape[0])
train_y = (train_y - np.mean(train_y)) / np.std(train_y)
train_y = train_y.reshape((-1, 1))

val_x = np.random.random((10000, 80))
val_y = np.sum(val_x[:, :39:2]**3, axis=1) * np.sum(val_x[:, 1:40:2]**2, axis=1) + np.sum(val_x[:, :40:4], axis=1) #+ np.random.random(val_x.shape[0])
val_y = (val_y - np.mean(val_y)) / np.std(val_y)
val_y = val_y.reshape((-1, 1))

test_x = np.random.random((10000, 80))
test_y = np.sum(test_x[:, :39:2]**3, axis=1) * np.sum(test_x[:, 1:40:2]**2, axis=1) + np.sum(test_x[:, :40:4], axis=1) #+ np.random.random(test_x.shape[0])
test_y = (test_y - np.mean(test_y)) / np.std(test_y)
test_y = test_y.reshape((-1, 1))

#####################pytorch code
import time
import re
import os
import logging
import torch
torch.set_num_threads(8)
import torch.nn as nn
import pickle
from collections import OrderedDict
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader, Sampler
import torch.nn.functional as F
LOG_FORMAT = '%(levelname)s %(asctime)s.%(msecs)03d %(filename)s[%(lineno)d]: %(message)s'
DATE_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(level = logging.INFO,
                    format=LOG_FORMAT,
                    datefmt=DATE_FORMAT)
logging.getLogger().setLevel('INFO')
logging.info(f"train_x[:5, :5]:\n{train_x[:5, :5]}")

class ModelDataset(Dataset):
    """docstring for ModelDataset"""

    def __init__(self, x, y, x_dtype=torch.float, y_dtype=torch.float):
        super(ModelDataset, self).__init__()
        self.x = torch.tensor(x, dtype=x_dtype)
        self.y = torch.tensor(y, dtype=y_dtype)

    def __len__(self):
        return self.x.shape[0]

    def __getitem__(self, i):
        return self.x[i], self.y[i]

class MLP(nn.Module):
    '''MLP
    input_dimention: 28
    结构 28*20*10*3
    '''

    def __init__(self, input_dim=26, dropout=0.5, hidden_layers=[128, 64]):
        super(MLP, self).__init__()
        self.dropout = dropout
        self.hidden_layers = hidden_layers
        self.model = nn.Sequential()
        layers = [input_dim] + hidden_layers
        for i in range(len(layers)-1):
            self.model.add_module(
                'fc%d' % (i+1), nn.Linear(layers[i], layers[i+1]))
            self.model.add_module('bn%d' % (i+1), nn.BatchNorm1d(layers[i+1]))
            self.model.add_module('dropout%d' % (i+1), nn.Dropout(dropout))
            self.model.add_module('relu%d' % (i+1), nn.ReLU())
        self.model.add_module('fc_out', nn.Linear(layers[-1], 1))

        for name, param in self.model.named_parameters():
            #bn层默认初始化
            if "bn" in name or "relu" in name:
                continue
            if "weight" in name:  # weight matrix
                nn.init.xavier_uniform_(param.data)
            else:  # bias
                nn.init.zeros_(param.data)
        # last layer: random normal
        for name, param in self.model.fc_out.named_parameters():
            #bn层默认初始化
            if "bn" in name or "relu" in name:
                continue
            if "weight" in name:  # weight matrix
                nn.init.normal_(param.data)
            else:  # bias
                nn.init.zeros_(param.data)

    def forward(self, input_data):
        '''input_data:  (x: batch_size*input_dim, '''
        output = self.model(input_data)
        return output

#params
save_dir = "/root/work"
init_lr = 0.005
l2_regularizer = 1e-5
loss = 'mse'
device = torch.device('cpu')
# device = torch.device("cuda:0")
np.random.seed(6)
torch.manual_seed(6)
torch.backends.cudnn.deterministic=True
torch.backends.cudnn.benchmark=True
model = MLP(train_x.shape[1], 0.5, [40, 20, 20])
model.to(device)
criterion = nn.MSELoss()
train_dataset = ModelDataset(train_x, train_y)
val_dataset = ModelDataset(val_x, val_y)
if loss == 'mse':
    criterion = nn.MSELoss()

# dataloader
train_loader = DataLoader(dataset=train_dataset, batch_size=512*4, shuffle=True, num_workers=4)
val_loader = DataLoader(dataset=val_dataset, batch_size=512*4, shuffle=False, num_workers=4)

#optimizer: only regularize weight matrix like keras
reg_params = []
no_reg_params = []
for name, param in model.named_parameters():
    match = re.match(r".*weight", name)
    if match is None or "bn" in name:
        no_reg_params.append(param)
    else:
        logging.info(f"regularizing: {name}")
        reg_params.append(param)

params = [
    {'params': reg_params, 'weight_decay': l2_regularizer, 'name': 'reg_params'},
    {'params': no_reg_params, 'weight_decay': 0, 'name': 'no_reg_params'}
]
optimizer = torch.optim.Adam(params, lr=init_lr)
# optimizer = torch.optim.SGD(params, lr=init_lr, momentum=0.9)
optim_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, 'min', factor=0.2, patience=6, cooldown=20)

##################################train code
max_epochs = 100
min_loss = np.inf # for checkpoint
for epoch in range(1, 1+max_epochs):
    epoch_start_time = time.time()
    for data, target in train_loader:
        pass
    for_loop_time = time.time() - epoch_start_time
    #train
    model.train()
    for data, target in train_loader:
        # pass
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
    logging.info(f"Epoch: {epoch} Training: loss: {loss.item():.4f}, LR: {optimizer.param_groups[0]['lr']:.5f}")

    #eval
    model.eval()
    with torch.no_grad():
        val_loss_sum = 0
        val_count = 0
        for data, target in val_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            loss = criterion(output, target)
            val_loss_sum += loss.item() * data.shape[0]
            val_count += data.shape[0]
        val_loss = val_loss_sum / val_count
        epoch_end_time = time.time()
        epoch_time = epoch_end_time - epoch_start_time
        logging.info(f"\t\tEpoch: {epoch} Validation: loss: {val_loss:.4f} using {epoch_time:.4f}s, loader loop {for_loop_time:.4f}s, training {epoch_time - for_loop_time:.4f}s")

    optim_scheduler.step(val_loss)
    #checkpoint
    if val_loss < min_loss:
        torch.save(model.state_dict(), os.path.join(save_dir, f'best_model.pth'))
        min_loss = val_loss

diff = np.ravel(model(torch.tensor(val_x, dtype=torch.float)).cpu().detach().numpy()) - np.ravel(val_y)
logging.info(f"val loss: {np.mean(diff**2)}")

diff = np.ravel(model(torch.tensor(test_x, dtype=torch.float)).cpu().detach().numpy()) - np.ravel(test_y)
logging.info(f"test loss: {np.mean(diff**2)}")

keras
#####################generate data
#f(x) = x[:39:2]**3 * x[1:40:2]**2 + x[:40:4] + noise
import numpy as np
seed = 6
np.random.seed(seed)
train_x = np.random.random((300000, 80))
train_y = np.sum(train_x[:, :39:2]**3, axis=1) * np.sum(train_x[:, 1:40:2]**2, axis=1) + np.sum(train_x[:, :40:4], axis=1) #+ np.random.random(train_x.shape[0])
train_y = (train_y - np.mean(train_y)) / np.std(train_y)
train_y = train_y.reshape((-1, 1))

val_x = np.random.random((10000, 80))
val_y = np.sum(val_x[:, :39:2]**3, axis=1) * np.sum(val_x[:, 1:40:2]**2, axis=1) + np.sum(val_x[:, :40:4], axis=1) #+ np.random.random(val_x.shape[0])
val_y = (val_y - np.mean(val_y)) / np.std(val_y)
val_y = val_y.reshape((-1, 1))

test_x = np.random.random((10000, 80))
test_y = np.sum(test_x[:, :39:2]**3, axis=1) * np.sum(test_x[:, 1:40:2]**2, axis=1) + np.sum(test_x[:, :40:4], axis=1) #+ np.random.random(test_x.shape[0])
test_y = (test_y - np.mean(test_y)) / np.std(test_y)
test_y = test_y.reshape((-1, 1))

#####################keras code
import os
import random
# from tensorflow.keras.callbacks import *
from tensorflow.keras import backend as K
import tensorflow as tf
from keras.models import Model
from keras import layers
from keras.models import Sequential
from keras.layers import Input, Dense, Dropout, Activation, noise, normalization
from keras.models import load_model
from keras import regularizers
from keras.optimizers import Adam, SGD
from keras.layers.normalization import BatchNormalization
from keras.callbacks import EarlyStopping
from keras.callbacks import ReduceLROnPlateau
from keras.callbacks import ModelCheckpoint
import logging
LOG_FORMAT = '%(levelname)s %(asctime)s.%(msecs)03d %(filename)s[%(lineno)d]: %(message)s'
DATE_FORMAT = '%Y-%m-%d %H:%M:%S'
logging.basicConfig(level = logging.INFO,
                    format=LOG_FORMAT,
                    datefmt=DATE_FORMAT)
logging.getLogger().setLevel('INFO')
logging.info(f"train_x[:5, :5]:\n{train_x[:5, :5]}")

seed_value = 6
os.environ['PYTHONHASHSEED']='0'
# os.environ["CUDA_VISIBLE_DEVICES"] = "1"
random.seed(seed_value)
np.random.seed(seed_value)
tf.random.set_seed(seed_value)

save_dir = "/root/work"
os.makedirs(save_dir, exist_ok=True)
inited_lr = 0.005
l1_regularizer = 0
l2_regularizer = 1e-5
hidden_layers = [40, 20, 20]
dropout = 0.5
model_k = Sequential()
model_k.add(Dense(hidden_layers[0], input_dim=train_x.shape[1], kernel_regularizer=regularizers.l2(l2_regularizer)))
model_k.add(BatchNormalization())
model_k.add(Dropout(dropout))
model_k.add(Activation('relu'))
for i in range(1,len(hidden_layers)):
    model_k.add(Dense(hidden_layers[i], input_dim=hidden_layers[i - 1], kernel_regularizer=regularizers.l2(l2_regularizer)))
    model_k.add(BatchNormalization())
    model_k.add(Dropout(dropout))
    model_k.add(Activation('relu'))
output_layer = Dense(1, input_dim=hidden_layers[-1], kernel_initializer='random_normal', kernel_regularizer=regularizers.l2(l2_regularizer))
model_k.add(output_layer)
model_k.compile(loss='mean_squared_error', optimizer=Adam(lr=inited_lr, beta_1=0.9, beta_2=0.999, decay = 0.0, epsilon=1e-08))
# model_k.compile(loss='mean_squared_error', optimizer=SGD(lr=inited_lr, momentum=0.9))
reduce_lr = ReduceLROnPlateau(monitor='val_loss', factor=0.2, patience=6, mode='auto', cooldown=20, verbose=1)
checkpoint = ModelCheckpoint(filepath=os.path.join(save_dir, 'best_model_k.h5'), monitor='val_loss',verbose=0, save_best_only=True, period=1)
#reduce_lr
hist = model_k.fit(train_x, train_y, epochs = 100, shuffle = True, batch_size = 512 * 4, validation_data=(val_x, val_y), callbacks=[reduce_lr, checkpoint])

diff = np.ravel(model_k.predict(val_x)) - np.ravel(val_y)
logging.info(f"val loss: {np.mean(diff**2)}")

diff = np.ravel(model_k.predict(test_x)) - np.ravel(test_y)
logging.info(f"test loss: {np.mean(diff**2)}")

Train Platform
##########pytorch
# (torch) ➜  dev git:(master) ✗ python3.8 -m pip show torch
# Name: torch
# Version: 1.7.0a0+df252c0
# Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration
# Home-page: https://pytorch.org/
# Author: PyTorch Team
# Author-email: packages@pytorch.org
# License: BSD-3
# Location: /root/miniconda3/envs/torch/lib/python3.8/site-packages
# Requires: future, numpy
# Required-by:

###########keras
#➜  quantum git:(dev) pip show keras
# Name: Keras
# Version: 2.4.3
# Summary: Deep Learning for humans
# Home-page: https://github.com/keras-team/keras
# Author: Francois Chollet
# Author-email: francois.chollet@gmail.com
# License: MIT
# Location: /usr/local/lib/python3.7/dist-packages
# Requires: pyyaml, scipy, numpy, h5py
# Required-by:

###########linux server
# (torch) ➜  work lscpu
# Architecture:        x86_64
# CPU op-mode(s):      32-bit, 64-bit
# Byte Order:          Little Endian
# CPU(s):              72
# On-line CPU(s) list: 0-71
# Thread(s) per core:  2
# Core(s) per socket:  18
# Socket(s):           2
# NUMA node(s):        2
# Vendor ID:           GenuineIntel
# CPU family:          6
# Model:               79
# Model name:          Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz
# Stepping:            1
# CPU MHz:             1244.824
# CPU max MHz:         3300.0000
# CPU min MHz:         1200.0000
# BogoMIPS:            4199.96
# Virtualization:      VT-x
# L1d cache:           32K
# L1i cache:           32K
# L2 cache:            256K
# L3 cache:            46080K

Context:
Updated
I found that in Keras, random_normal initializer initalize the weights with Normal(0, 0.05) not Normal(0, 1). So I changed the pytorch last layer weights initialization to
nn.init.normal_(param.data, std=std)  results becomes better. I was surprised that in such a simple task, weights initialization takes huge effect. I changed the random seed and do 24 experiments, Keras is not superior to Pytorch anymore, it’s all about randomness(玄学)
Screen Shot 2020-11-17 at 1.23.55 PM1346×1728 384 KB


Question:
Hi,
The net parameters(weights and network models) should be stored and then load. But When I try to load it I got an Error. Could I change the following code in a way it works?
net = models.vgg16_bn().to(device)
Fix_net_par1 = net.named_parameters()
Fix_net_par = copy.deepcopy(dict(Fix_net_par1))
net.load_state_dict(Fix_net_par)       <=== The line that creates the Error

RuntimeError: Error(s) in loading state_dict for VGG:
Missing key(s) in state_dict: “features.1.running_mean”, “features.1.running_var”, “features.4.running_mean”, “features.4.running_var”, “features.8.running_mean”, “features.8.running_var”, “features.11.running_mean”, “features.11.running_var”, “features.15.running_mean”, “features.15.running_var”, “features.18.running_mean”, “features.18.running_var”, “features.21.running_mean”, “features.21.running_var”, “features.25.running_mean”, “features.25.running_var”, “features.28.running_mean”, “features.28.running_var”, “features.31.running_mean”, “features.31.running_var”, “features.35.running_mean”, “features.35.running_var”, “features.38.running_mean”, “features.38.running_var”, “features.41.running_mean”, “features.41.running_var”.
Could I change the above code in a way it works?

Context:
Hello,
I never stored and reloaded parameter runtime.
Anyway why you don’t just save your model parameters on your HD and then reload as indicated in the PyTorch tutorials saving_loading_models_tutorial 6
Alessandro.


Question:
I’m experiencing an intermittent issue with my GeForce 3090. The issue was occurring with nightly before 1.7 was released, and with the 1.7 version. It was also occurring with the nvidia driver 455.23 and after updating to 455.32.
For the release version, after installing 455.32, I installed PyTorch with:
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch

Sometimes the python process hangs after say the ten thousandth iteration or the 5 millionth. The issue occurs regardless the network that I am training (GAN, CNN, etc).
I don’t know that this is a PyTorch issue, but I am hopeful for suggestions on how to investigate what might be causing the hang so that I may report the issue to the appropriately. Any suggestions would be appreciated.

Context:
Are you using a single GPU or multiple ones?
Could you kill the process when it hangs and post the stack trace here, please?


Huge gpu performance difference between using NVIDIA RXT 2070 and GTX 1080 GPUs

Question: Even though these two GPUs are somewhat close in terms of compute specifications. the GTX 1080 being slightly better. When I train my model on a GTX 1080 GPU powered machine, it takes 0.5 seconds of GPU processing time for a single batch. Whereas an RTX 2070 powered machine takes 9 seconds in average for the same operation. This happens for a variety of models I have trained including pure CNN, CNN+LSTM, CNN+RNN. I am also using the cudnn.benchmark = True and pin_memory = True, for both machines. Both machines have roughly the same CPU and memory capabilities. Any thoughts as to why the RTX machine could be 18x longer for the same operation?

Context:
Did you make sure to synchronize the code properly before starting and stopping the timer?
Since CUDA operations are asynchronous, you’ll get wrong profiling results without synchronizations.
Also, are you using the same PyTorch, CUDA and cudnn versions?


Question:
Hello, I have a densenet model and I want to run it in the browser. Does anyone know any solutions I can use?
I tried onnx.js and I get the following error:
Uncaught (in promise) TypeError: cannot resolve operator 'Pad' with opsets: ai.onnx v9

Context:
Hi. I don’t know if you have a custom densenet model but assuming that you are using a pre-trained model from torchvision library, then you can use the following steps for running your model in the browser:

Using ONNX/ONNX JS:

import torch
import torch.onnx
import torchvision.models as models

# Load the pre-trained model
densenet = models.densenet161(pretrained=True)

# Dummy Input Data
dummy_input = torch.rand(1, 3, 256, 256)

# Define input / output names
input_names = ["features/conv0"]
output_names = ["classifier"]

# Convert PyTorch to ONNX
torch.onnx.export(densenet,
                  dummy_input,
                  "densenet_op11.onnx",
                  verbose=True,
                  input_names=input_names,
                  output_names=output_names,
                  opset_version= 11)

Now once you have the model in ONNX format, you can use this tutorial to serve the model: https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html 5

Using TensorFlow.js for model serving

import torch
import torch.onnx
import torchvision.models as models
import tensorflow as tf

# Load Pre-trained Model
densenet = models.densenet161(pretrained=True)

# Dummy Input Data
dummy_input = torch.rand(1, 3, 256, 256)

# Define input / output names
input_names = ["features/conv0"]
output_names = ["classifier"]

# Convert PyTorch to ONNX
torch.onnx.export(densenet,
                  dummy_input,
                  "densenet_op11.onnx",
                  verbose=True,
                  input_names=input_names,
                  output_names=output_names,
                  opset_version= 11)

Once you have the ONNX model, run this command in command line:
# Onnx to TensorFlow
#  TensorFlow = 2.3.0
# Install onnx-tf from here: https://github.com/onnx/onnx-tensorflow
# pip install tensorflow-addons

onnx-tf convert -i ./densenet_op11.onnx -o ./densenet_op11.pb

Once you have the TF Model, convert that to TFjs using the following command on terminal:
# Convert TensorFlow to TFjs
# https://github.com/tensorflow/tfjs/tree/master/tfjs-converter

tensorflowjs_wizard

In the above, select the model type as TensorFlow Saved Model and provide the rest of the things like model save location and all.
Screen Shot 2020-11-05 at 7.39.18 PM1085×417 73.2 KB
Once you have the TFjs model, you can run inference using this in the browser. For some examples, look here: https://github.com/tensorflow/tfjs-examples 3
I hope this helps


Question:
I am trying to deploy my flask application using Docker, but before running it in a container, I would like to verify that it works in a linux environment.
After installing all the packages in the right versions (pytorch 1.6, torchtext 0.7.0) as I did in windows, I get the following error for my exported model:
untimeError: Error(s) in loading state_dict for RNN:
        size mismatch for embedding.weight: copying a param with shape torch.Size([101616, 100]) from checkpoint, the shape in current model is torch.Size([100944, 100]).

How can i fix this?

Context:
One nn.Embedding layer changed its size and thus the state_dict cannot be loaded.
Check, how the num_embeddings are calculated on your systems and make sure to use the same value (e.g. 101616 from the checkpoint).
Often these errors between Windows and Linux happen, because your code might read/sort/find different files or files in the wrong order etc., as these system calls behave differently on these OSes.


How to calculate the memory requirement for running inferences

Question:
Hi Everyone,
I have trained a deep learning model for image classification.
The trained model size is 250 Mb. How to calculate, how much computation memory, it’ll require to run inferences.
I am trying to run it on my 16GB RAM Macbook Pro, but it is failing after the first iteration.

Context:
Hi. I think you can use the PyTorch Profiler here: https://pytorch.org/tutorials/recipes/recipes/profiler.html 7   for getting the amount of memory required per layer, execution time per layer in the model as well as show the amount of memory (used by the model’s tensors) that was allocated (or released) during the execution of the model’s operators.
Hope this helps


Question:
I’m not sure this is a bug or not.
I need to deploy the AWS elastic inference for our service. The Elastic Inference requires using CPU to load and run models.
but our code runs well on GPUs, but CPU.
as the simple code below, it can be run on GPUs but on CPU it returns “index out of range in self” Error.

CPUs returns index out of range in self error
import numpy as np
import torch
import torch.nn as nn
sinusoid_table = torch.FloatTensor(torch.Size([50 + 1, 512]))
pos_emb = nn.Embedding.from_pretrained(sinusoid_table, freeze=True)
positions = torch.arange(200).expand(1, 200).contiguous()+1
positions=positions
a= pos_emb(positions)
print(a)
on GPUs this run well
import torch
import torch.nn as nn
device = torch.device(‘cuda:0’)
sinusoid_table = torch.FloatTensor(torch.Size([50 + 1, 512])).to(device)
pos_emb = nn.Embedding.from_pretrained(sinusoid_table, freeze=True).to(device)
positions = torch.arange(200).expand(1, 200).contiguous()+1
positions=positions.to(device)
a= pos_emb(positions)
print(a)

I’m using PytorchECL 1.3.1 (support Elastic Inference) comes with Amazon AMI
Tested with normal pytorch==1.5.0
both of them result the “index out of range error”
Any helps is appreciated. thank you.

Context:
The code raises an error on the CPU as well as the GPU using the latest stable version (1.7.0).
Note that assert statements in CUDA code were mostly disabled in 1.5.0 due to a bug, so that your code doesn’t raise the proper error if you are using the GPU.
The reason for the error is that your weight lookup table in the embedding layer contains feature vectors for 51 indices, so your inputs are limited to [0, 50], while you are passing positions with values in the range [1, 200].
Also, note that torch.FloatTensor(torch.Size([51, 512])) will create a tensor in the specified shape with uninitialized values, which might contain invalid values such as NaNs and Infs, so you should properly initialize sinusoid_table or use a tensor factory method to create it such as torch.randn.


How to take inputs through pre-trained CNNs, one layer at time?

Question:
Hello - I am new to pytorch & these forums, apologies if I haven’t directed this question to the right place.  Before I begin, I am using Python 3.7 and torch==1.5.0 incase either of these matter.
I am struggling to take inputs forward through an arbitrary pre-trained CNN, one layer/module at a time. I understand that forward hooks can be used to capture intermediate features. What I am hoping to find is a generalized way to replay intermediate features through subsequent layers, without requiring a full forward pass.  Is there a best practice for this in PyTorch?
[edit] perhaps simplest form of my question is how to call
x = net.features(x)
y = net.classifier(x)

When I call all net.children() in sequence I either get an output shape that does not match what’s produced by full forward pass, or else execution errors passing intermediate features to the next layer.  Calling net.features() followed by net.classifier() also produces same/similar size mismatch errors described above.
I am not looking for fix specific to densenet161, but use it here as an example:
import torch
import torchvision.models as tmodels
net = tmodels.densenet161(pretrained=True)
net.eval()

x = torch.randn(1, 3, 227, 227)
y = net(x).data.numpy()
print('y full:', y.shape)

for child in net.children():
  x = child(x)
y = x.data.numpy()
print('y part:', y.shape)

With this I get size mismatch errors in the net.children() loop:
$ python ./test.py
y full: (1, 1000)
Traceback (most recent call last):
  File "./test.py", line 11, in <module>
    x = child(x)
  File "/Users/mn/miniconda3/envs/env/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/mn/miniconda3/envs/env/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/Users/mn/miniconda3/envs/env/lib/python3.7/site-packages/torch/nn/functional.py", line 1612, in linear
    output = input.matmul(weight.t())
RuntimeError: size mismatch, m1: [15456 x 7], m2: [2208 x 1000] at ../aten/src/TH/generic/THTensorMath.cpp:41

Insights/help much appreciated.  Thanks!

Context:
You could apply all child modules, if the model would generally work in an nn.Sequential container.
The model would thus have to define only nn.Modules in its __init__ method and apply these modules sequentially.
However, this is often not the case and the forward method often uses functional calls such as x.view() etc., especially if the model architecture is more complicated then a simple stack of layers.
The densenet you are using is for example applying these calls from the functional API, which are lost if you just try to call the modules.


Question:
I met this warning when converting CRNN to ONNX model, my code is as follows:
from torch import nn,onnx
import torch

class BidirectionalLSTM(nn.Module):

    def __init__(self, nIn, nHidden, nOut):
        super(BidirectionalLSTM, self).__init__()

        self.rnn = nn.LSTM(nIn, nHidden, bidirectional=True)
        self.embedding = nn.Linear(nHidden * 2, nOut)

    def forward(self, input):
        recurrent, _ = self.rnn(input)
        T, b, h = recurrent.size()
        t_rec = recurrent.view(T * b, h)
        output = self.embedding(t_rec)  # [T * b, nOut]
        output = output.view(T, b, -1)
        return output

class CRNN(nn.Module):

    def __init__(self, imgH, nc, nclass, nh, n_rnn=2, leakyRelu=False, lstmFlag=True):
        """
        是否加入lstm特征层
        """
        super(CRNN, self).__init__()
        assert imgH % 16 == 0, 'imgH has to be a multiple of 16'

        ks = [3, 3, 3, 3, 3, 3, 2]
        ps = [1, 1, 1, 1, 1, 1, 0]
        ss = [1, 1, 1, 1, 1, 1, 1]
        nm = [64, 128, 256, 256, 512, 512, 512]
        self.lstmFlag = lstmFlag

        cnn = nn.Sequential()

        def convRelu(i, batchNormalization=False):
            nIn = nc if i == 0 else nm[i - 1]
            nOut = nm[i]
            cnn.add_module('conv{0}'.format(i),
                           nn.Conv2d(nIn, nOut, ks[i], ss[i], ps[i]))
            if batchNormalization:
                cnn.add_module('batchnorm{0}'.format(i), nn.BatchNorm2d(nOut))
            if leakyRelu:
                cnn.add_module('relu{0}'.format(i),
                               nn.LeakyReLU(0.2, inplace=True))
            else:
                cnn.add_module('relu{0}'.format(i), nn.ReLU(True))

        convRelu(0)
        cnn.add_module('pooling{0}'.format(0), nn.MaxPool2d(2, 2))  # 64x16x64
        convRelu(1)
        cnn.add_module('pooling{0}'.format(1), nn.MaxPool2d(2, 2))  # 128x8x32
        convRelu(2, True)
        convRelu(3)
        cnn.add_module('pooling{0}'.format(2),
                       nn.MaxPool2d((2, 2), (2, 1), (0, 1)))  # 256x4x16
        convRelu(4, True)
        convRelu(5)
        cnn.add_module('pooling{0}'.format(3),
                       nn.MaxPool2d((2, 2), (2, 1), (0, 1)))  # 512x2x16
        convRelu(6, True)  # 512x1x16

        self.cnn = cnn
        if self.lstmFlag:
            self.rnn = nn.Sequential(
                BidirectionalLSTM(512, nh, nh),
                BidirectionalLSTM(nh, nh, nclass))
        else:
            self.linear = nn.Linear(nh * 2, nclass)

    def forward(self, input):
        # conv features
        conv = self.cnn(input)
        b, c, h, w = conv.size()
        conv = conv.squeeze(2)
        conv = conv.permute(2, 0, 1)  # [w, b, c]
        output = self.rnn(conv)
        return output

model = CRNN(32, 1, 5530, 256,n_rnn=2,leakyRelu=False,lstmFlag=True)
x = torch.rand(1,1,32,100)

onnx.export(
    model,x,"crnn.onnx",export_params=True,opset_version=11,# verbose=True,
    do_constant_folding=True,input_names=["input"],output_names=["output"],
    dynamic_axes={'input' : {0 : 'batch_size'},'output' : {1 : 'batch_size'}}
)

The full information of warning is :
/home/dai/py36env/lib/python3.6/site-packages/torch/onnx/symbolic_opset9.py:1377: UserWarning: Exporting a model to ONNX with a batch_size other than 1, with a variable lenght with LSTM can cause an error when running the ONNX model with a different batch size. Make sure to save the model with a batch size of 1, or define the initial states (h0/c0) as inputs of the model.
  "or define the initial states (h0/c0) as inputs of the model. ")

How can I avoid this warning or how to define the initial states(h0/c0)？

Context: hi,Have you found a solution for this problem?


Question:
Grettings.
Suppose I have a model and a certain data. I train the model on the data, now I want to create two independent copies of said trained model. With independent I mean that changes done to one copy of the model should not affect the other copy, or the original model.
I tried to save the trained model with torch.save(model, PATH) and then simply model_copy = torch.load(PATH). But this ends up that these two are “connected”. Is my programming off or why is this the case?
Best regards
Erik`

Context:
How about doing this:
import copy

net_a = torch.load(PATH)
net_b = copy.deepcopy(net_a)


Question:
Hi everyone, I have developed a deep-learning object-detect program based on pytorch and it works very well. Today I deploy this program on a PC, everything goes well, but the program cannot be launched. Debug and find out that, the program blocks, or halts, on importing pytorch.
Simply start a python prompt, type import torch and the prompt blocks. top command shows that CPU/memory usage is very low. Press ctrl-c cannot stop the prompt. While other library importing is fine. I have tried pycrypto and the one I wrote myself, all work but pytorch cannot.
I have deployed more than 100 times, but never meet this situation. I also tried to reinstall pytorch, from 1.6 to 1.4, torchvision from 0.7 to 0.5, still not work. No error printed, no complain shown.
Environment:

OS: centos 7.4
CUDA: 10.0
NVIDIA driver: 440.82
GPU: GTX 1660
python: 3.6

pytorch version: 1.6 and 1.4.
Any information are welcome, thanks in advance.

Context:
Did you deploy on this setup and system before?
If so, what could have changed between the successful deployment and the issue now, i.e. did you change something in the OS/system?
Also, could you check if you are able to run a simple PyTorch script in a docker container?


Not all model parameters are transfered to GPU

Question:
I’m noticing that if I run model.to(device), the tensors used in the forward are not necessarily transferred over the GPU. Below is a minimal example of this
import torch
from torch.distributions import MultivariateNormal

class SimpleModel(torch.nn.Module):
    def __init__(self, q):
        super(SimpleModel, self).__init__()
        self._zero_mean = torch.zeros(2 * q)
        self._eye_covar = torch.eye(2 * q)
        self.mvn = MultivariateNormal(self._zero_mean, self._eye_covar)
    def reparameterize(self, mu, logv):
        eps = torch.randn_like(mu)
        z = mu + eps * torch.exp(logv)
        logp = self.mvn.log_prob(eps)
        return z, logp
    def forward(self, x):
        return self.reparameterize(x, x)

model = SimpleModel(10).to('cuda')
x = torch.ones(20).to('cuda')
model(x)

This gives the following error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-bd0200007a4a> in <module>
----> 1 model(x)

~/miniconda3/envs/mavi/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-10-7cdded9571ea> in forward(self, x)
     14         return z, logp
     15     def forward(self, x):
---> 16         return self.reparameterize(x, x)

<ipython-input-10-7cdded9571ea> in reparameterize(self, mu, logv)
     11         eps = torch.randn_like(mu)
     12         z = mu + eps * torch.exp(logv)
---> 13         logp = self.mvn.log_prob(eps)
     14         return z, logp
     15     def forward(self, x):

~/miniconda3/envs/mavi/lib/python3.8/site-packages/torch/distributions/multivariate_normal.py in log_prob(self, value)
    205         if self._validate_args:
    206             self._validate_sample(value)
--> 207         diff = value - self.loc
    208         M = _batch_mahalanobis(self._unbroadcasted_scale_tril, diff)
    209         half_log_det = self._unbroadcasted_scale_tril.diagonal(dim1=-2, dim2=-1).log().sum(-1)

RuntimeError: expected device cuda:0 but got device cpu

I know that I can manually override the to() and cuda() methods, so there is a solution. But I’m curious why the to() and cuda() can’t already handle these sorts of edge cases (or if there is something that I am overlooking).

Context:
Hi
Yes I don’t think the distribution objects are moved around by the nn.Modules.
But If you want _zero_mean and _eye_covar to be moved around, you need to register them as buffers in the nn.Module.


Question:
is there any way to export the actual computation graph of a model that is trained or not? Since ‘model.state_dict()’ give the weights in an ordered dict. By some manual coding, It can be made as JSON. Theoretically, weights can be sent as JSON over the net. If it possible to generate sent computation graph as follows
class linearRegression(nn.Module):
    def __init__(self):
        super(linearRegression, self).__init__()
        self.linear = nn.Linear(1, 1)

    def forward(self, x):
        out = self.linear(x)
        return out

model = linearRegression()

model.state_dict()

#OrderedDict([('linear.weight', tensor([[0.6289]])),
#            ('linear.bias', tensor([0.7113]))])

but when I have to export to libtorch(C++), I have to make an exact same copy of the class to use the model. I guess there might be an easy way which I don’t know that to an export graph from different language

Context: You can use the JIT and script the model as described here 33.


Question:
Hello, I am trying to get pytorch running inside docker, but not having much luck. Here is some terminal output first from the host machine, then immediately after running inside of a base nvidia/cuda docker image:
$ nvidia-smi
...
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
...
$ nvidia-docker run --rm -it nvidia/cuda:10.2-base-ubuntu18.04 /bin/bash
# nvidia-smi
...
| NVIDIA-SMI 440.44       Driver Version: 440.44       CUDA Version: 10.2     |
...
# apt-get update && apt-get install -y python3 python3-pip && pip3 install torch torchvision
...
# python3 -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.6.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: Ubuntu 18.04.5 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: No
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GeForce RTX 2080 SUPER
Nvidia driver version: 440.44
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.19.2
[pip3] torch==1.6.0
[pip3] torchvision==0.7.0
[conda] Could not collect

I am using prepackaged nvidia/cuda docker containers and a minimal python install, so I’m not sure what could be going wrong. All of the problems I have found other people have so far have caused nvidia-smi to not work properly, but in my case, it is working fine, yet torch still says CUDA is not available.

Context: I wiped the OS, re-installed everything from scratch and it is now working.


How to load an older model in PyTorch 1.6?

Question:
I have some models trained in PyTorch 1.5 and saved by torch.save(filename, model), instead of torch.save(filename, model.state_dict()).
Now, I am not able to load them in PyTorch 1.6.
I get torch.nn.modules.module.ModuleAttributeError: 'BatchNorm2d' object has no attribute '_non_persistent_buffers_set'
What is the correct way to load such models?
Following https://github.com/ultralytics/yolov5/issues/58 14, I tried this on the model and seemed to load without problem. But is this the right way?
model = ModelDenseNet()
model1 = torch.load(model_file)
for k, m in model1.named_modules():
   m._non_persistent_buffers_set = set()  # pytorch 1.6.0 compatability
model.load_state_dict(model1.state_dict())

Thanks

Context:
blackberry:

Would be great if PyTorch provided and option to torch.save/torch.load to save load either the whole model or the state_dict.

The recommended way of storing the state_dict should be working fine or are you seeing any issues with it? As explained in the Serialization semantics 23, storing and loading the model directly could be breaking in various ways due to the way Python pickles the model.


Question:
Hi i’m trying to convert this model to onnx.
class onnx_Autoencoder(nn.Module):
 def __init__(self):
        super(onnx_Autoencoder, self).__init__()

        self.Cov1=nn.Conv2d(1,32,3)
        self.batchNorm1=nn.BatchNorm2d(32)
        self.relu1 =nn.ReLU()

        self.Cov2 = nn.Conv2d(32, 64, 3)
        self.batchNorm2 = nn.BatchNorm2d(64)
        self.relu2 = nn.ReLU()

        #
        self.Transpose1= nn.ConvTranspose2d(64, 32, 3)
        self.batchNorm1Tra=nn.BatchNorm2d(32)
        self.relu1Tra=nn.ReLU(inplace=True)

        self.Transpose2 = nn.ConvTranspose2d(32, 1, 3)

Everything is working fine except when i add track_running_stats=False in batch norm layer

self.batchNorm1=nn.BatchNorm2d(32,track_running_stats=False)

When i check the exported onnx model i get the following errror
Traceback (most recent call last):
  File "onnx_check_model.py", line 5, in <module>
    onnx.checker.check_model(onnx_model)
  File "/home/velab/anaconda3/lib/python3.7/site-packages/onnx/checker.py", line 86, in check_model
    C.check_model(model.SerializeToString())
onnx.onnx_cpp2py_export.checker.ValidationError: Node ()'s input 3 is marked single but has an empty string in the graph

==> Context: Bad node spec: input: "27" input: "batchNorm1Tra.weight" input: "batchNorm1Tra.bias" input: "" input: "" output: "30" output: "31" output: "32" output: "batch_norm_dead_output-72" output: "batch_norm_dead_output-73" op_type: "BatchNormalization" attribute { name: "epsilon" f: 1e-05 type: FLOAT } attribute { name: "momentum" f: 0.9 type: FLOAT }

Could someone provide me an idea on how to solve this?
Thanks

Context: I have the same issue. Any new findings to solve this problem?


How to do ONNX to TensorRT in INT8 mode?

Question:
Hello.
I am working with the subject, PyTorch to TensorRT.
With a tutorial, I could simply finish the process PyTorch to ONNX.
And, I also completed ONNX to TensorRT in fp16 mode.
However, I couldn’t take a step for ONNX to TensorRT in int8 mode.
Debugger always say that `You need to do calibration for int8*.*
Does anyone know how to do convert ONNX model to TensorRT int8 mode?
Thank you in adavance

Context:
This thread says exporting quantized pytorch models to onnx is not yet supported.

    ONNX export of quantized model quantization

    General export of quantized models to ONNX isn’t currently supported. We only support conversion to ONNX for Caffe2 backend. This thread has additional context on what we currently support - ONNX export of quantized model

You can try quantizing after you export pytorch model to onnx by using onnxruntime 49.
Otherwise, you may want to check out if direct export from pytorch to tensorrt supports quantized models.

      GitHub

NVIDIA-AI-IOT/torch2trt 131

An easy to use PyTorch to TensorRT converter. Contribute to NVIDIA-AI-IOT/torch2trt development by creating an account on GitHub.


CXXABI_* and GLIBCXX_* not found after building pre-abi Libtorch from source using gcc 5.4.0

Question:
Hi,
After building the pre-abi libtorch from source using gcc 5.4.0,
using the following steps
conda activate pytorch-build

# Setup compiler gcc 5.4.0
export CC=~/gcc-5.4.0/bin/gcc
export CXX=~/gcc-5.4.0/bin/g++
export LD_LIBRARY_PATH_BACKUP=$LD_LIBRARY_PATH
export LD_LIBRARY_PATH=~/gcc-5.4.0/lib64:$LD_LIBRARY_PATH

# Pytorch env variables
export BUILD_CAFFE2_MOBILE=OFF BUILD_TEST=OFF USE_OPENMP=0 USE_CUDA=0 USE_CUDNN=0 USE_MKLDNN=0 USE_MKL=0 USE_NNPACK=0 USE_QNNPACK=0 USE_DISTRIBUTED=0 USE_FFMPEG=0 USE_FBGEMM=0 USE_PYTORCH_QNNPACK=0
export PYTORCH_BUILD_VERSION=1.6.0 PYTORCH_BUILD_NUMBER=0
export _GLIBCXX_USE_CXX11_ABI=0

# Build libtorch
cd ~/Repos/pytorch
git checkout v1.6.0
python setup.py clean
mkdir -p build_libtorch && cd build_libtorch
python ../tools/build_libtorch.py

# Disable gcc 5.4.0 to use 4.8.2
unset CC && unset CXX && export LD_LIBRARY_PATH=$LD_LIBRARY_PATH_BACKUP

Then after switching to an older gcc 4.8.2, the command ldd libtorch.so throws some errors :
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by /.../pytorch/build_libtorch/build/lib/libtorch_cpu.so)
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /.../pytorch/build_libtorch/build/lib/libtorch_cpu.so)
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /.../pytorch/build_libtorch/build/lib/libtorch_cpu.so)
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /.../pytorch/build_libtorch/build/lib/libtorch_cpu.so)
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by /.../pytorch/build_libtorch/build/lib/libc10.so)
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by /.../pytorch/build_libtorch/build/lib/libc10.so)
build/lib/libtorch.so: /.../gcc/4.8.2/lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /.../pytorch/build_libtorch/build/lib/libc10.so)

I also tried with no success to hardcode the value of D_GLIBCXX_USE_CXX11_ABI to 0 in the pytorch/CMakeLists.txt:
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -D_GLIBCXX_USE_CXX11_ABI=0")
# or
add_definitions(-D_GLIBCXX_USE_CXX11_ABI=0)

OS used: Centos 7
The distribution pre-abi provided by pytorch: https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-1.6.0%2Bcpu.zip doesn’t have this issue.
Any idea how to solve this ?
Regards,

Context: Solution: https://github.com/pytorch/pytorch/issues/45064#issuecomment-696466277 31


Why can’t self defined layers run on GPUs

Question:
20200919_0840281148×896 77.9 KB
20200919_084106593×852 64.4 KB
def forward( self, x ):
start = time.time()
out = self.Rconv2d( x )
cost = time.time()-start
out = self.maxpool1( out )
out = self.block1( out )
out = self.block2( out )
out = self.block3( out )
out = F.avg_pool2d( out, 7 )
out = out.view( out.size(0), -1 )
out = self.out_layer( out )
total_cost = time.time()-start
print(cost/total_cost)
return out
最后使用time模块查看该层执行效率，发现这个变换层模块占了模型99.9%的时间

Context:
From Google translate:

Finally, use the time module to check the execution efficiency of this layer and find that this transformation layer module accounts for 99.9% of the model’s time

Could you please use an online translate service, so that we could support you?
Custom layers can run on the GPU. However, since CUDA operations are asynchronous, you would have to synchronize the code via torch.cuda.synchronize() before starting and stopping the timer.
Also, please post code snippets by wrapping them into three backticks ```, which makes debugging easier.


Question:
Hello
I’m a student who studies deep learning with Pytorch.
I have a question, but I want to learn only a specific character dataset (for example, alphabet V) when learning EMNIST dataset with Pytorch, but I want to know how to implement this with code.
Thanks!

Context:
You could manipulate the targets such that your target class uses the class index 1 while all other classes are assigned the class index 0.
Afterwards you can treat it as a binary classification and could use e.g. nn.BCEWithLogitsLoss as your criterion.


Question:
I installed CUDA 10.1.105 and reinstall NVIDIA driver 430.64 , cuDnn 7.6.5 even Anaconda because of no available
python -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.3.1
Is debug build: No
CUDA used to build PyTorch: Could not collect

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: No
CUDA runtime version: 10.1.105
GPU models and configuration: GPU 0: GeForce RTX 2080 Ti
Nvidia driver version: 430.64
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.5

Versions of relevant libraries:
[pip3] numpy==1.19.1
[pip3] torch==1.3.1
[pip3] torchvision==0.4.2

But till
>>> import torch
>>> print(torch.cuda.is_available())
False
>>> exit()

Please, I need help ,  solution or sth

Context:
You don’t need local CUDA and cudnn installations, if you are using the conda binary.
Only the NVIDIA driver will be used.
Based on the output you are seeing, I assume you’ve installed the CPU binary.
Could you reinstall PyTorch with the desired CUDA version using the install command from here 126 and post the install log, if it’s still not working?


Dynamic Input for ONNX.js using a Pytorch trained model

Question:
So I’ve got this autoencoder that I’ve trained and now I wanna deploy it to a website.
I came across ONNX.js and liked it because it can run natively on the browser(Saving me my much needed Student cloud credits )
However I noticed that onnx requires a dummy input so that it can trace the graph and this requires a fixed input size.
dummy = torch.randn(1, 3, 1920, 1080)
torch.onnx.export(transformer, dummy, save_path, opset_version=11)

I’m almost certain that on the ONNX.js side I need the input to be the same shape.
Is there anyway around this, because I need dynamic shape inputs.
Or should I just used Flask and host the model?
Thanks!

Context:
I’m by no means an expert, but I think you can use the dynamic_axes optional argument to onnx.export
In the tutorial here 43 (about a quarter of the way down) the example uses the dynamic_axes argument to have a dynamic batch size:
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable lenght axes
                                'output' : {0 : 'batch_size'}})

I assume that can also be done for the other axes, but I haven’t tried it - my use case has images which are guaranteed to be the same size, so I don’t need it.
I have seen some discussion that would suggest this is not the case though, so maybe someone with more experience can weigh in?


Question:
Question about onnx exporting a model which uses torch.rfft() in the forward function. How would this work as ONNX opset doesn’t support FFT yet.
Do let me know. Thanks!

Context:
I think you’ll just get a funny-looking operator (aten::…) in ONNX.
The other question is, of course, which backend would than execute your operator.
Best regards
Thomas


Pytorch inference is slow inside Flask API

Question:
I encounter a problem with the inference speed between inference in main thread and inside Flask API enpoint (which is inside another thread). The inference speed is about 10 times slower.
Do you have any idea about this problem and any solution?

Context:
Do you run your PyTorch model in Python or have you JITed it to TorchScript?
We discuss this a bit in Chapter 15 of our book 45, if you run your model through the JIT - even from Python - you avoid the infamous Python GIL for all intermediates, which may give you an edge in multithreaded deployment scenarios.
Best regards
Thomas


Question:
Hi it’s a small problem I am facing with respect to model saving and loading.
I am training a simple classification model in Google Colab. As per the instructions from https://pytorch.org/tutorials/recipes/recipes/saving_and_loading_models_for_inference.html 4
I am saving the model as
torch.save(net.cpu().state_dict(),"/folder/model.pt")

After this, I am downloading the saved file from colab. While trying to load it(ONLY FOR INFERENCE)in my local machine as
model = NetClass()
model.load_state_dict(torch.load("/local_folder/model.pt")

It does not load the model but returns an error as
RuntimeError: ./mymodel/model.pt is a zip archive (did you mean to use torch.jit.load()?)

I tried unzipping the downloaded file manually and it has a directory structure which is expected. Now I don’t know where to proceed.
I tried the same method in the same colab file itself and IT WORKED. But in local file it is NOT WORKING
Any kind of help and suggestions would be highly appreciated!
Thank you!

Context:
Are you using different PyTorch versions in Colab and on your local machine?
If so, I guess your local installation might be older so could you update it?


How to add pytorch as a requirement for a windows python package

Question:
I have a python package which depends on pytorch and which I’d like windows users to be able to install via pip (the specific package is: https://github.com/mindsdb/lightwood 2, but I don’t think this is very relevant to my question).
What are the best practices for going about this ?
Are there some project I could use as examples ?
It seems like the pypi hosted version of torch & torchvision aren’t windows compatible and the “getting started” section suggests installing from the custom pytorch repository, but beyond that I’m not sure what the ideal solution would be to incorporate this as part of a setup script.

Context: Pytorch is available on windows. did you check Pytorch.org 1?


Cannot load Sentiment model for prediction

Question:
I have a Sentiment model trained using glove vectors refer https://blog.floydhub.com/long-short-term-memory-from-zero-to-hero-with-pytorch/ 1.
I save the trained mode and dictionary
torch.save(trainedmodel, ‘./trained_lstm.pt’)
torch.save(trainedmodel.state_dict(), ‘./state_dict.pt’)
I try to recreate the model for inference
model = torch.load(‘trained_lstm.pt’, map_location=torch.device(“cpu”))
but keep getting error cannot get attribute SentimentNet in module main.  If I just load the state dictionary i get error on embedding and weight sizes.
I included the model class in the same file while saving and reloading. Any help ?

Context:
I would recommend to stick to the second approach (save and load the state_dict), as the former approach can break in various ways.
That being said, could you post the error message for the shape mismatch and check, what might have changed in the model?
E.g. did you store the state_dict and changed the embedding layer afterwards somehow?


Deploying a pytorch model in meteor + react js

Question:
I have gone through a couple of pages looking at my options.I intend to build a Pytorch model, I already have a web app all written in meteor+react. I am not sure if there is a way to combine flask and meteor. Else, is how easy is it to deploy my model for both torch and TF, then use Tensorflow.js  with my react +meteor app.
Would appreciate any advice. i have most of my front end build already so that stack can’t change now.

Context:
Hi Moseli!
Hope you’re going good.
Given you already have a reactjs frontend, you probably want to maintain the separation of Backend and frontend. I think in your case, it would make sense in your case to build a Flask API, and have your reactjs app simply call your external API?
I wrote an answer back here, in Sep 2019 50 pointing to links. Since then, PyTorch has implemented TorchServing which could be of interest - blog here 54, docs here. 19


“AttributeError: ‘ReLU’ object has no attribute ‘threshold’ ”

Question:
Hi, All.
I have met this error when run pytorch inference in Jetson nano.
"AttributeError: ‘ReLU’ object has no attribute ‘threshold’ ".
What wrong to me?
Jetson nano, Ubuntu 18.04LTS aarch64, Pytorch 1.1.0, Torchvision 0.3.0
Thanks.
@bemoregt.

Context: Could you post the code snippet which raises this error?


Question:
If performance is the priority here, would it be okay to deploy a heavy model for action recognition natively on mobile (using pytorch mobile)?
Or should I go with something like rest api, hosted on other remote server (amazon, azure, etc).

Context:
Having it on remote is more logical unless you have very bad internet connection.
its not customary to place the actual model on the device, as it puts everything into risk.