Deploying multiple PyTorch models on a single GPU

Question:
Hello PyTorch community,
Suppose I have 10 different PyTorch models (classification, detection, embedding) and 10 GPUs. I would like to serve real-time image traffic on these models. We can assume a uniform traffic distribution for each model. What is the most efficient (low latency, high throughput) way?

Deploy all 10 models onto each and every GPU (10 models on all 10 GPUs). This will probably incur context switch and cache miss costs. Memory management might also be costlier. Latency will be high. (?)
Deploy a single model on each GPU (10 models on 10 GPUs). No context switch or cache miss costs. Can be good for uniform traffic. (?)

Any suggestions, insights, experience?
Thanks!

Context:
I can tell that someone was watching Tesla AI day.
You can deploy as many models as your gpu has memory to hold. However, during training, a model can require 4-5 times more gpu ram and equivalent increase in calculation time as your model is doing optimization. So you might be better allocating GPUs accordingly via a distributed training workflow. But make sure the size of all your GPUs can fit on the deployment GPU. One way to do this would be to test the 10 models on your GPU untrained and in eval mode. Granted, the inference wouldn’t be any good as you haven’t trained them, yet, but it will show if you need to tune model sizes accordingly. Then you can tweak model sizes before training. Make sure you have a memory buffer of 10-15% from the max memory.


How to make the best use of a single GPU with a small model

Question:
I have a small model that uses up very little of the GPUs resources. I am looking for the best method to run data through this model as quickly as possible. The order of the data does not matter so no synchronization needs to occur.
I have looked into multiprocessing, however I have been unable to pass the model/tensors as .cuda(), and converting them every time seems to be a significant slowdown.
What is the best way to approach the problem with getting as much out of a single GPU as possible.

Context: High batch size should work. Batch size parallelizes compute so as to maximize GPU cores usage. You could think of a batch size of 1,000 as running 1,000 models side by side to compute the results. Granted, that is a subpar comparison because once you maximize your cores usage, higher batch size won’t reduce total job speed.


Question: Can we Infere 2 model at the same time on a GPU with Torchserve? For example, I want to Infere a model 1 and 2 on the same system at the same time. How can I do that?

Context:
Here is a workflow for multiple models on TorchServe.

      github.com

    serve/examples/Workflows at master · pytorch/serve 1

  master/examples/Workflows

  Model Serving on PyTorch. Contribute to pytorch/serve development by creating an account on GitHub.


Question:
Good day!
I have created a new docker image that is based on pytorch/torchserve:0.4.2-cpu. It includes some additional dependencies (most importantly a modified version of fairseq), and an entrypoint script that downloads the Model Archive (.mar) file from AWS S3 and places it in /home/model-server/model-store/ then starts torchserve, like so:
    torchserve --start --ts-config /secrets/config.properties

When I run this docker image on my development machine, everything works great. The .mar file is downloaded from S3, Torchserve starts up, loads the model, and successfully responds to prediction requests.
However, when I try to deploy this docker image in Hashicorp’s Nomad Orchestration System, I see the following issue. The .mar file is successfully downloaded from S3, Torchserve starts up successfully, but then as it’s loading the model, it blows up without any helpful error (that I can see).
Here is the output that I see in the logs for the nomad allocation:
2021-09-30 19:34:50,836 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager - Initializing plugins manager...
2021-09-30 19:34:50,999 [INFO ] main org.pytorch.serve.ModelServer -
Torchserve version: 0.4.2
TS Home: /usr/local/lib/python3.6/dist-packages
Current directory: /home/model-server
Temp directory: /home/model-server/tmp
Number of GPUs: 0
Number of CPUs: 4
Max heap size: 512 M
Python executable: /usr/bin/python3
Config file: /secrets/config.properties
Inference address: http://0.0.0.0:8080
Management address: http://0.0.0.0:8081
Metrics address: http://0.0.0.0:8082
Model Store: /home/model-server/model-store
Initial Models: en-de.mar
Log dir: /home/model-server/logs
Metrics dir: /home/model-server/logs
Netty threads: 32
Netty client threads: 0
Default workers per model: 4
Blacklist Regex: N/A
Maximum Response Size: 6553500
Maximum Request Size: 6553500
Prefer direct buffer: false
Allowed Urls: [file://.*|http(s)?://.*]
Custom python dependency for model allowed: true
Metrics report format: prometheus
Enable metrics API: true
Workflow Store: /home/model-server/model-store
Model config: {"en-de": {"0.0.1": {"defaultVersion": true,"marName": "en-de.mar","minWorkers": 1,"maxWorkers": 1,"responseTimeout": 5000}}}
2021-09-30 19:34:51,019 [INFO ] main org.pytorch.serve.servingsdk.impl.PluginsManager -  Loading snapshot serializer plugin...
2021-09-30 19:34:51,021 [INFO ] main org.pytorch.serve.ModelServer - Loading initial models: en-de.mar
2021-09-30 19:35:09,606 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Adding new version 0.0.1 for model en-de
2021-09-30 19:35:09,606 [DEBUG] main org.pytorch.serve.wlm.ModelVersionedRefs - Setting default version to 0.0.1 for model en-de
2021-09-30 19:35:19,441 [INFO ] main org.pytorch.serve.wlm.ModelManager - Model en-de loaded.
2021-09-30 19:35:19,442 [DEBUG] main org.pytorch.serve.wlm.ModelManager - updateModel: en-de, count: 1
2021-09-30 19:35:19,455 [INFO ] main org.pytorch.serve.ModelServer - Initialize Inference server with: EpollServerSocketChannel.
2021-09-30 19:35:19,527 [INFO ] main org.pytorch.serve.ModelServer - Inference API bind to: http://0.0.0.0:8080
2021-09-30 19:35:19,527 [INFO ] main org.pytorch.serve.ModelServer - Initialize Management server with: EpollServerSocketChannel.
2021-09-30 19:35:19,528 [INFO ] main org.pytorch.serve.ModelServer - Management API bind to: http://0.0.0.0:8081
2021-09-30 19:35:19,528 [INFO ] main org.pytorch.serve.ModelServer - Initialize Metrics server with: EpollServerSocketChannel.
2021-09-30 19:35:19,529 [INFO ] main org.pytorch.serve.ModelServer - Metrics API bind to: http://0.0.0.0:8082
Model server started.
2021-09-30 19:35:19,741 [WARN ] pool-2-thread-1 org.pytorch.serve.metrics.MetricCollector - worker pid is not available yet.
2021-09-30 19:35:19,822 [INFO ] pool-2-thread-1 TS_METRICS - CPUUtilization.Percent:0.0|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:19,823 [INFO ] pool-2-thread-1 TS_METRICS - DiskAvailable.Gigabytes:150.82331466674805|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:19,823 [INFO ] pool-2-thread-1 TS_METRICS - DiskUsage.Gigabytes:56.5672492980957|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:19,823 [INFO ] pool-2-thread-1 TS_METRICS - DiskUtilization.Percent:27.3|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:19,824 [INFO ] pool-2-thread-1 TS_METRICS - MemoryAvailable.Megabytes:70207.8671875|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:19,824 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUsed.Megabytes:24832.73828125|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:19,824 [INFO ] pool-2-thread-1 TS_METRICS - MemoryUtilization.Percent:26.3|#Level:Host|#hostname:a6c81b0f4290,timestamp:1633030519
2021-09-30 19:35:20,182 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - Listening on port: /home/model-server/tmp/.ts.sock.9000
2021-09-30 19:35:20,182 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - [PID]105
2021-09-30 19:35:20,182 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - Torch worker started.
2021-09-30 19:35:20,182 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - Python runtime: 3.6.9
2021-09-30 19:35:20,183 [DEBUG] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerThread - W-9000-en-de_0.0.1 State change null -> WORKER_STARTED
2021-09-30 19:35:20,187 [INFO ] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerThread - Connecting to: /home/model-server/tmp/.ts.sock.9000
2021-09-30 19:35:20,197 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - Connection accepted: /home/model-server/tmp/.ts.sock.9000.
2021-09-30 19:35:20,236 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - model_name: en-de, batchSize: 1
2021-09-30 19:35:21,193 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - Sentencepiece model path: /home/model-server/tmp/models/0b779a55a9e84e42996d07ab39e39a31/spm.model
2021-09-30 19:35:24,147 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - [en] dictionary: 32000 types
2021-09-30 19:35:24,147 [INFO ] W-9000-en-de_0.0.1-stdout MODEL_LOG - [de] dictionary: 32000 types
2021-09-30 19:35:25,304 [INFO ] epollEventLoopGroup-3-1 ACCESS_LOG - /10.136.1.138:16775 "GET /models HTTP/1.1" 200 4
2021-09-30 19:35:25,304 [INFO ] epollEventLoopGroup-3-1 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:Unknown,timestamp:null
2021-09-30 19:35:26,570 [INFO ] epollEventLoopGroup-4-1 ACCESS_LOG - /10.136.1.138:49125 "GET /metrics HTTP/1.1" 200 1
2021-09-30 19:35:26,571 [INFO ] epollEventLoopGroup-4-1 TS_METRICS - Requests2XX.Count:1|#Level:Host|#hostname:Unknown,timestamp:null
2021-09-30 19:35:27,779 [INFO ] W-9000-en-de_0.0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-en-de_0.0.1-stdout
2021-09-30 19:35:27,779 [INFO ] W-9000-en-de_0.0.1-stderr org.pytorch.serve.wlm.WorkerLifeCycle - Stopped Scanner - W-9000-en-de_0.0.1-stderr
2021-09-30 19:35:27,780 [INFO ] epollEventLoopGroup-5-1 org.pytorch.serve.wlm.WorkerThread - 9000 Worker disconnected. WORKER_STARTED
2021-09-30 19:35:27,780 [DEBUG] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerThread - System state is : WORKER_STARTED
2021-09-30 19:35:27,781 [DEBUG] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerThread - Backend worker monitoring thread interrupted or backend worker process died.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
	at java.base/java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:432)
	at org.pytorch.serve.wlm.WorkerThread.run(WorkerThread.java:188)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
2021-09-30 19:35:27,782 [WARN ] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.BatchAggregator - Load model failed: en-de, error: Worker died.
2021-09-30 19:35:27,782 [DEBUG] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerThread - W-9000-en-de_0.0.1 State change WORKER_STARTED -> WORKER_STOPPED
2021-09-30 19:35:27,783 [WARN ] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-en-de_0.0.1-stderr
2021-09-30 19:35:27,783 [WARN ] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerLifeCycle - terminateIOStreams() threadName=W-9000-en-de_0.0.1-stdout
2021-09-30 19:35:27,783 [INFO ] W-9000-en-de_0.0.1 org.pytorch.serve.wlm.WorkerThread - Retry worker: 9000 in 1 seconds.

Any idea what might be going on? Or any ways to get additional information about the failure?
Thanks in advance!

Context:
Despite sizing the Nomad allocation’s memory limit to allow for about 2x the size of the loaded model, it seems that during some initialization steps, the allocation required a bit more than that amount of memory, which was causing it to silently fail.
Increasing the memory limit to 3x the size of the loaded model made this work.


Export object detection model to ONNX:empty output by ONNX inference

Question:
I try to convert my PyTorch object detection model (Faster R-CNN) to ONNX. I have two setups. The first one is working correctly but I want to use the second one for deployment reasons. The difference lies in the example image which I use for the export of the function torch.onnx.export().
In the first setup I use a real image as input for the ONNX export. But in a official tutorial 8 they say that I can use a dummy input, which should have the same size as the model expects the input. So I created a tensor with the same shape but with random values. The export in both setups is working correctly. But the second setup does not deliver the desired results after inference with the ONNX runtime. The code and the exemplary output can be found below.
Setup 1
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)
...
checkpoint = torch.load(model_state_dict_path)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

to_tensor = transforms.ToTensor()
img_rgb = Image.open(image_path_model).convert('RGB')
img_rgb = to_tensor(img_rgb)
img_rgb.unsqueeze_(0)

torch.onnx.export(model, img_rgb, "detection.onnx", opset_version=11)

I get no error and the export works. Afterwards I run the model with the ONNX runtime and I get the following output:
[array([[704.0696  , 535.19556 , 944.8986  , 786.1619  ],
         ...], dtype=float32),
array([2, 2, 2, 2, 2, 1, 1], dtype=int64),
array([0.9994363 , 0.9984769 , 0.99816966, ...], dtype=float32)]

The output is as I expect it to be (Bounding boxes, object classes and probabilities).
Setup 2
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained = True)
...
checkpoint = torch.load(model_state_dict_path)
model.load_state_dict(checkpoint['model_state_dict'])
model.eval()

img_rgb = torch.randn(1, 3, 1024, 1024)

torch.onnx.export(model, img_rgb, "detection.onnx", opset_version=11)

Like in setup 1 I get no error and the export works. Afterwards I run the model with the ONNX runtime and with the same image as in setup 1 and I get the following output:
[array([], shape=(0, 4), dtype=float32),
array([], dtype=int64),
array([], dtype=float32)]

It is just an empty array.
What is wrong with the second setup? I am new to ONNX. The export runs the model. Do I have to provide an input on which the model also recognizes objects and therefore the dummy input with random values does not work? Is the statement “The values in this can be random as long as it is the right type and size.” only valid for the provided tutorial?

Context: Related: https://github.com/pytorch/vision/issues/1706#issuecomment-658198204 198


Question:
Hi Everyone,
I have trained a model on pytorch.
Now, I want to design a complete pipeline around it. Has anyone tried Tensorflow extended with it and can comment on the compatibility issues with it(if any).
Also, any other suggestions would also be welcomed regarding the production and the complete pipeline.

Context: Would be interested to hear an answer to the OP’s question or if anyone knows if the PyTorch team is working on a similar end-to-end ML ops solution like TFX?


How to deploy a Text to Speech model on AWS

Question:
Hello,
I’m trying to deploy my Trained Text to speech model using AWS lambda services but as I haven’t worked on the deployment part before so I’m having trouble to figure out the initial steps. My model consist of two trained models one is waveglow for inferencing and the other one is my trained tacotron2 model on a specific language. I know how to create a lambda function and trigger s3 bucket in which my model is saved but cant figure out how to define a lambda function. Please guide me with the process. Thanks in advance.

Context: Hey, did you ever solve your problem? Which tutorial or options did you try? I’m curious if you tried using sagemaker for model hosting since you’re already on AWS, such as described here: Deploy a Trained PyTorch Model — Amazon SageMaker Examples 1.0.0 documentation 3.


What is the minimum cuda environment for pytorch

Question:
pytorch1.7.1+cuda11.0

If I want to run pytorch with CUDA, what is the minimum cuda environment? (C++/CUDA extension is not required). Should I install all CUDA libraries? (cuNPP, cuSPARSE, cuBLAS, NCCL etc.)
And, Does the torch-1.7.1+cu110-cp36-cp36m-linux_x86_64.whl contains a cuDNN lib? Should I install an additional package If I want to use cuDNN in pytorch?

Context: You would only need to install an NVIDIA driver to run PyTorch if you install the pip wheels or conda binaries, as they ship with the CUDA runtime, cuDNN, NCCL, cublas, etc.


How to customize data types that can be transferred between CPU and GPU

Question:
When I design my network I need to work with “complex” data whose leaf attributes are tensors. For example, CameraSpecification contains attributes:

position, a tensor
transform, a tensor
intrinsics,  data of another type CameraIntrinsics containing:

fov, a tensor
pixel_coordinates, a tensor

Then these two classes are nothing but groups of tensors, so in theory they can be safely transferred between CPU and GPU and can be generated during forward(). However, if I generate them during forward(), I will have to move the tensor attributes of them using to(). This is especially inconvenient when I use PytorchLightning. Is there a more elegant way to group tensors and to group groups of tensors? Thank you!

Context:
I’m not sure, if the posted object is used as the input or inside the model as e.g. a custom parameter.
In the former case, I would guess you could push the data to the GPU before executing the forward pass, while the parameters would be pushed to the device after initializing the model, so I’m unsure why the device transfer is used in the forward.
In any case, I don’t know what limitations Lightning has, so could you explain why pushing the data to the device won’t work?


TensorRT 8.0: Compiling pytorch 1.8 with CUDA11.3

Question:
Hello,
I’m trying to setup TensorRT 8.0, the Jetson TX2 I’m using uses ARM, hence I need to run TensorRT8.0+CUDA11.3. To make sure training and deployment have similar environment, I followed the documentation for TensorRT setup 1 and downloaded CUDA11.3.
Now I’m trying to get pytorch 1.8, given pytorch 1.8 is compiled with CUDA 11.1, I’m trying to compile myself. I got the pytorch 1.8 from: pytorch/releases/tag/v1.8.1 and is trying to follow this guide 3 to compile. However, I can’t get pass the ‘git submodule update --init --recursive’, it says ‘fatal: not a git repository’
Hence:

Should i be following this guide at all? Given im compiling version 1.8
If ‘Yes’ to 1, how do I circumvent the ‘fatal: not a git repository’ error?
if ‘No’ to 1, what should I be doing to compile pytorch 1.8?

Thanks

Context:
I can’t put more than 2 links
I got pytorch 1.8 from: Release Small bug fix release · pytorch/pytorch · GitHub 2


How to store model in memory when serving/deploy

Question: I have this trained model (jit traced) now, and yes I can serve it using flask etc… but, every time the inference is done, the model is loaded each time - I would like to avoid this model load every time and keep this in memory somehow. What are the suggested approaches for this? I could not find any on pytorch site.

Context:
Hello @John_J_Watson,
Perhaps you have already checked these links: DEPLOYING PYTORCH IN PYTHON VIA A REST API WITH FLASK 1and PyTorch Flask API. Especially, the second one shows that the load_model function can be written somewhere else, not in the script where flask lauches directly.
So the flask won’t reload the model again and again.
Hope it helps somehow.
Jian


Best practice to handle batches caching with GPU

Question:
Hi I am learning the best ways to manage batches and other best practices during model training and inference and I have some:

If I have a batch that I move to GPU, should. I move it back to CPU after doing training? if no why?
batch, label = batch.to(device), label.to(device)
model(batch)
..Training pass...
batch, label = batch.cpu(), label.cpu()

If I cache my data in my Dataset class how can I ensure I can reuse the same batches on gpu to avoid transferring from and to cpu multiple times?

Is to a best practice to calculate all performance metrics with detached tensors to avoid storing other graph nodes in memory?

Thanks in advance for any possible help.

Context:
Usually you would neither move the batch back to the CPU not cache it, as often you are applying random transformations on-the-fly. However, if your use case is different, you could of course try to store the data in a manual cache and try to reuse it. In this case, moving it back to the CPU would save GPU memory and would most likely be needed (unless you are working with a tiny dataset and model and you GPU is able to store both).

Yes, I would recommend to detach() the tensors just in case 3rd party libraries would store the tensors in any way, which could increase the GPU memory since the entire computation graph would also be stored. Often you are calculating the metrics from the validation set, so you could also wrap the entire validation processing in a with torch.no_grad() block.


ONNX model inference produces different results for the idential input

Question:
After a Pytorch model is converted to ONNX and then loaded in onnxruntime.
For the idential input, it produces different results in ONNX model. (works in PyTorch)
Please refer to this reproducible notebook

      colab.research.google.com

Google Colaboratory 5

Context: I worked out the issue being the use of PReLU, not sure about the fix yet.


Does the official conda install include / require cudnn?

Question:
I notice that the official conda install doesn’t include cudnn. Is that required separately? Or it is included in the cudatoolkit?
FYI the website says to use:
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

Context: cuDNN is statically linked into PyTorch, as no conda binaries or pip wheels are available for it.


TypeError: cross_entropy_loss(): argument ‘input’ (position 1) must be Tensor, not tuple

Question:
def __init__(self):
        super(CNNLenet, self).__init__()
        self.conv1 = nn. Sequential(
            nn.Conv2d(1,16, 5,1,2),
            nn.ReLU(),
            nn.MaxPool2d(2)
            )
        self.conv2 = nn.Sequential(
            nn.Conv2d(16, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2)
        )
        # fully connected layer, output 10 classes
        self.fc1 = nn.Linear(32 * 7 * 7, 10)
    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        # flatten the output of conv2 to (batch_size, 32 * 7 * 7)
        x = x.view(x.size(0), -1)
        output = self.fc1(x)
        return output, x    # return x for visualization

Here is optimization loop

```for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)

        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

``` ---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-23-458279d9b112> in <module>
     11         # forward + backward + optimize
     12         outputs = model(inputs)
---> 13         loss = criterion(outputs, labels)
     14
     15         loss.backward()

~\anaconda3\envs\deeplearning\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
   1049         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1050                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1051             return forward_call(*input, **kwargs)
   1052         # Do not call functions when jit is used
   1053         full_backward_hooks, non_full_backward_hooks = [], []

~\anaconda3\envs\deeplearning\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
   1118
   1119     def forward(self, input: Tensor, target: Tensor) -> Tensor:
-> 1120         return F.cross_entropy(input, target, weight=self.weight,
   1121                                ignore_index=self.ignore_index, reduction=self.reduction)
   1122

~\anaconda3\envs\deeplearning\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2822     if size_average is not None or reduce is not None:
   2823         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2824     return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2825
   2826

TypeError: cross_entropy_loss(): argument 'input' (position 1) must be Tensor, not tuple

I am unable to understand this error. Kindly help me in understanding and resolving it.

Context:
The forward method of your model returns a tuple via:
return output, x    # return x for visualization

which creates the issue in loss = criterion(outputs, labels).
I assume you want to use output to calculate the loss, so use:
        output, x = model(inputs)
        loss = criterion(output, labels)

and it should work.


Memory leakage during realtime inference of pytorch_pretrained_bert’s BertForSequenceClassification model

Question:
0
I’m facing memory leakage in the realtime inference of pytorch_pretrained_bert’s BertForSequenceClassification model.
Although I’m using GPU but still CPU memory is exhausting
      with torch.no_grad():
         logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask)
         logits = logits.detach().cpu().numpy()
         predictions.append(logits)
         gc.collect()
         torch.cuda.empty_cache()
         del logits

model, b_input_ids and b_input_mask are mapped on gpu using .to(device)
Can anyone suggest me what can I do… memory is increasing gradually

Context:
Ashbajawed:

Although I’m using GPU but still CPU memory is exhausting

I see in the code you are appending the logits to predictions which will increase the cpu memory.


Packaging pytorch topology first and checkpoints later

Question:
In 2019-2020 I wrote a library similar to the current incarnation of torch.package, called torch_liberator (GitHub - Kitware/torch_liberator 1). The key difference is that torch-liberator relies on static analysis (via a separate library I wrote called liberator: GitHub - Kitware/liberator).
The way I used it was: a model class (or a pointer to the file that contained the class and the class name) would be passed to the export_model_code function. This would start a new empty file. First, it would extract the code for that class only, and then, based on what the user specified in “export_modules” (analogous to “intern”-ing in torch.package), it would iteratively (and statically) pull in dependencies from other modules. The result of this process was python code that could be exported.
The next thing that the user specfied were the keyword arguments that could be used to construct an instance of the exported class. If possible these were saved as Python literals, but if not the offending kwargs were pickled. This minimized the use of pickling, which I view as a desirable property. Lastly, the user specified an existing snapshot on-disk. These three pieces (1) the model code (2) the instantiation kwargs, and (3) the model state were packged into a zipfile. In production this exported module could be loaded more-or-less independently  of the training code, an instance of the class could be created, and then the saved state could be loaded into the model.
This worked well, but it had a few problems. First, because this worked completely by static analysis, it was fragile to non-basic ways of using it. Second, it didn’t munge any names except for the root package name. The new torch.package module in 1.9 solves a lot of these problems, and I want to move away from torch-liberator and start using torch.package.

This leads me to my question. It seems that torch.package.PackageExporter needs to be passed a model that has already been loaded with the desired model_state, because it needs to pickle the entire model class as export time.
I was wondering if there was a way to “prepare” a basic torch package on-disk, such that it only contained the model (perhaps with some random weight initialization, although ideally this wouldn’t be required to save space). Then given this exported model topology and some existing on-disk model_state checkpoint, I would like to simply add the model state checkpoint into the zipfile (perhaps overwriting some other file in the zipfile).
The use case is that at the start of training, I would like to export the untrained topology into the training directory. Then as model-state checkpoints are saved as training progresses, I would like to be able to quickly construct a torch package corresponding to that checkpoint state without having to load the whole thing into Python memory and then pickle it out again.
Is this at all possible with the current state of torch.package? Would it be possible to modify it to accomidate this use-case? Is this use case clear and reasonable?

Context:
I’d like to clarify the use case.
At the end of training I my trainer will save the best N candidate checkpoints based on training and validation metrics. I want to create a torch package for each of these N checkpoints to create N corresonding candidate packages that can then be evaluated in standalone prediction+evaluation code.
I could do this by looping over each checkpoint, and calling model.load_state_dict(torch.load(checkpoint_path)), but  this seems heavy-handed and unnecessary. The call to torch.load is expensive and calling model.load_state_dict is not thread safe, and has the side effect of modifying the current state of the model (and I would like to avoid side-effects if possible).
Is there a way to structure a torch.package, such that I can make a copy of some “base” zipfile (where ideally the purpose of that zipfile is to only store the network topology, perhaps it has some dummy file representing where the weight checkpoint would go, but it would be nice if that was kept small to minimize the time it takes to make a copy of the file) and then simply modify the copy by adding the desired checkpoint containing the pickled model_state_dict?


Question:
Older GPUs don’t seem to support torch in spite of recent cuda versions.
In my case the crash has the following error:
/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/cuda/__init__.py:83: UserWarning:
    Found GPU%d %s which is of cuda capability %d.%d.
    PyTorch no longer supports this GPU because it is too old.
    The minimum cuda capability supported by this library is %d.%d.

  warnings.warn(old_gpu_warn.format(d, name, major, minor, min_arch // 10, min_arch % 10))
WARNING:lightwood-16979:Exception: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1. when training model: <lightwood.model.neural.Neural object at 0x7f9c34df1e80>
Process LearnProcess-1:13:
Traceback (most recent call last):
  File "/home/maxs/dev/mdb/venv38/sources/lightwood/lightwood/model/helpers/default_net.py", line 59, in forward
    output = self.net(input)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/container.py", line 139, in forward
    input = module(input)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 96, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/maxs/dev/mdb/venv38/lib/python3.8/site-packages/torch/nn/functional.py", line 1847, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

This happens in spite of:
assert torch.cuda.is_available() == True
torch.version.cuda == '10.2'

How can I check for an older GPU that doesn’t support torch without actually try/catching a tensor-to-gpu transfer? The transfer initializes cuda, which wastes like 2GB of memory, something I can’t afford since I’d be running this check in dozens of processes, all of which would then waste 2GB of memory extra due to the initialization.

Context: Since the answer here might not be very torch-related and instead implicate usage of e.g. nvidia specific tools I also asked here: python - How to check torch gpu compatibility without initializing CUDA? - Stack Overflow 17 | Putting in the link in case anyone stumbles upon this question later and finds the SO answers useful.


Convert existing model/weights fromData Parallel to generic?

Question:
Sorry if this isn’t the correct topic. I did try to look for my problem, but I don’t I think I’m using the right search terms, which is why I’m here.
So, I trained a model using nn.DataParallel and it worked fine. I was able to inference fine on my machine, but only my machine. I finally figured out that I mistakenly did NOT save my model as generic as described here: https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-torch-nn-dataparallel-models
Is there a way to convert my existing model to a generic form so it can be used by anyone? Hopefully in a automated way and not me sludging through the weights manually?

Context:
Are you still able to load the checkpoint? DataParallel just hides the actual model away under the attribute module. So you can do something like this:
my_model = ModelClass(...)
my_model = DataParallel(my_model, ...)

state_dict = torch.load(...)
my_model.load_state_dict(state_dict)

actual_model = my_model.module
new_state_dict = actual_model

Haven’t tested this, but I bet something like this would work as well:
state_dict = torch.load(...)

new_state_dict = {}
for k, v in state_dict.items():
    k = k.replace('module.', '')  # Each weight k will be prefix with the word "module"
    new_state_dict[k] = v

model = SomeModel(...)
model.load_state_dict(new_state_dict)


How to drop layers from a custom PyTorch model

Question:
I have a custom model that I’m using, from this git repo 1
Here is the basic architecture (there are some additional conv layers):
  (lconv6aa): Sequential(
    (0): Conv2d(128, 196, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (lconv6a): Sequential(
    (0): Conv2d(196, 196, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (lconv6b): Sequential(
    (0): Conv2d(196, 196, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (corr): Correlation()
  (leakyRELU): LeakyReLU(negative_slope=0.1)
  (conv6_0): Sequential(
    (0): Conv2d(81, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (conv6_1): Sequential(
    (0): Conv2d(209, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (conv6_2): Sequential(
    (0): Conv2d(337, 96, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (conv6_3): Sequential(
    (0): Conv2d(433, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (conv6_4): Sequential(
    (0): Conv2d(497, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): LeakyReLU(negative_slope=0.1)
  )
  (fc1): Linear(in_features=529, out_features=512, bias=True)
  (fc1_trasl): Linear(in_features=512, out_features=256, bias=True)
  (fc1_rot): Linear(in_features=512, out_features=256, bias=True)
  (fc2_trasl): Linear(in_features=256, out_features=3, bias=True)
  (fc2_rot): Linear(in_features=256, out_features=4, bias=True)
  (dropout): Dropout(p=0.0)

The model is running with pre-trained weights, and I want to replace these last fc layers with my own.  Is there some way I can drop those layers, and replace it with my own custom ones?  I’m a beginner in pytorch and wasn’t able to figure this out based on the posts that I found here.  Any help would be appreciated

Context:
Yes, you can directly assign new modules to the model via e.g.:
model.fc1 = nn.Linear(...)

to replace the linear layer assigned to fc1.


Question:
Running the example from the pytorch website gives me this error. see: torch.linalg.cholesky — PyTorch 1.9.0 documentation 8
I’ve checked that all packages are updated.
!pip install torch -U --user
import torch
a = torch.randn(2, 2, dtype=torch.complex128)
a = a @ a.t().conj()  + torch.eye(2) # creates a Hermitian positive-definite matrix
l = torch.linalg.cholesky(a)
a
l
l @ l.t().conj()

AttributeError                            Traceback (most recent call last)
 in
1 a = torch.randn(2, 2, dtype=torch.complex128)
2 a = a @ a.t().conj()  + torch.eye(2) # creates a Hermitian positive-definite matrix
----> 3 l = torch.linalg.cholesky(a)
4 a
5 l
AttributeError: module ‘torch.linalg’ has no attribute ‘cholesky’

Context: I cannot reproduce the issue. Could you check the installed PyTorch version via torch.__version__, please?


Conv2D padding in TensorFlow and PyTorch

Question:
I am trying to convert TensorFlow model to PyTorch but having trouble with padding. My code for for relevant platforms are as follow:
TensorFlow
conv1 = tf.layers.conv2d(
                inputs=input_layer,
                filters=32,
                kernel_size=[5, 5],
                padding="same",
                activation=tf.nn.relu,
                name = "conv1")

PyTorch
conv1 = nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2)

I have few questions:

Are the above codes equivalent?
How many paddings are added on left/right/top/bottom if we use same padding in tensorflow?
How many paddings are added on left/right/top/bottom if we use padding=2 in pytorch?
If the above two code snippets are not equivalent then how can we make the same conv layer?

Thanks in advance.

Context:
Not entirely, since the TF implementation seems to add a relu to the conv layer while this is not the case in PyTorch.
I guess 2 on each side, but you would have to confirm it in TF.
2 on each side.


How to train two identical models in pytorch?

Question:
Hi,
I have a question about training in pytorch.
There is a pseudo code that I wanna ask.
model1 = Net()
model2 = copy.deepcopy(model1)

optimizer1 = torch.optim(model1.parameters(), lr)
optimizer2 = torch.optim(model2.parameters(), lr)

for i in range(dataset):

    x = random(dataset[i])
    y = dataset[i]

    update_step1(model1, optimizer1, x, y)
    update_step2(model2, optimizer2, x, y)

I thought this would be updated separately in each function(update_step1, 2), but the two values were the same.
How should I do to train model at the same time?
Note that the reason for doing this at the same time is to give the same random X value both models.
Thank you.

Context:
Your code should work and both models as well as optimizers would be independent.
However, if you don’t have any randomness in the models and are also feeding the same data as well as target into them, the training might be equal (unless non-deterministic operations are used internally with an undefined order of ops).
Here is a small example:
def check_models(model1, model2):
    for (name1, param1), (name2, param2) in zip(model1.named_parameters(),
                                                model2.named_parameters()):
        print('name {}, is equal {}'.format(
            name1, torch.allclose(param1, param2)))

def update_step(model, optimizer, x, y):
    optimizer.zero_grad()
    output = model(x)
    loss = F.mse_loss(output, y)
    loss.backward()
    optimizer.step()

model1 = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU(),
    #nn.Dropout(),
    nn.Linear(10, 10)
)
model2 = copy.deepcopy(model1)

check_models(model1, model2)

lr = 1e-3
optimizer1 = torch.optim.SGD(model1.parameters(), lr)
optimizer2 = torch.optim.SGD(model2.parameters(), lr)

for i in range(2):
    x = torch.randn(10, 10)
    y = torch.randn(10, 10)

    update_step(model1, optimizer1, x, y)
    update_step(model2, optimizer2, x, y)
    check_models(model1, model2)

As you can see, the updated parameters will be the same unless I enable the dropout layer in the model.


怎么创建一个跟随模型变更设备的张量？How to creat a Tensor with the same device as the model？

Question:
怎么在nn.Module里面，创建一个张量，他的设备跟随这一个模型改动？
比如说，我需要一个卷积核K，他参数是多少我给定的，不需要参与训练。但是我forward的时候需要他进行几次卷积。除了这一个K以外还有其他的卷积层。
如果model().cuda()，这个张量也是在GPU上，如果.cpu()，这个张量也在CPU上。
I wanna a Tensor created in nn.Module, it has the same device as the Module.
If model().cuda(), the Tensor is also moved to GPU, and if model.cpu(), it is moved to CPU

Context:
From Google translate:
Title:

How to create a tensor that follows the model change device?

Post:

How to create a tensor in nn.Module, and his equipment changes with this model?
For example, I need a convolution kernel K, the parameters of which are given by me, and I don’t need to participate in training. But when I forward, I need him to perform several convolutions. In addition to this K, there are other convolutional layers.
If model().cuda(), this tensor is also on the GPU, if .cpu(), this tensor is also on the CPU.

Please use an online translation service before posting the question here, so that all users could help you.


Sequential inference of multiple models in a memory limited environment

Question:
Dear fellow PyTorch users, I have a question concerning deployment in memory-constrained environments.
I wish to deploy a large number (more than 20) of models on the same data. For example, one model would detect the cars in the image and a separate network would segment the streets, another would apply domain transfer to improve the model performance of the other networks if the given scene has unusual characteristics (rain, low lighting, etc.).
Though it may be better to have a single model perform these tasks, I have found that using separate models gives much better performance. Also, our clients (and I myself) value the modularity that using separate modules provides.
Currently, using Tensorflow in graph mode provides automatic memory optimizations in GPU and CPU RAM memory. Even if there are 20 models, with 1GB of parameters in each model, the GPU memory allocated is not 20GB because Tensorflow automatically loads only the necessary models from disk to RAM and GPU DRAM. This saves on both GPU memory and RAM, both of which are valuable resources in deployment.
Is there any way to do something similar in PyTorch? I would like to run multiple, potentially very heavy, models in sequence on the same data without exhausting my GPU and RAM memory limitations. I have considered manually loading and sending each model in sequence. However, this would lead to inefficiencies if disk IO or CPU/GPU transfer were a bottleneck. Also, TorchScript models seem to discourage transferring the model after scripting is complete.
I have considered converting to ONNX. However, ONNX supports a relatively limited set of PyTorch operators and has numerous implementation differences that may lead to different outputs. I would like to ask if there is a PyTorch native solution to this problem.
Many thanks in advance for any replies.

Context:
Good day vertias,
without knowing details about the exact memory footprint, architecture and payload of inferences, it is not easy to give detailed help, but:

Have you taken a look at the mobile Interpreter 2? This may reduce the memory footprint by up to 75%.
Consider spending more time on producing an end-to-end architecture, since this will save all your problems in one go. A good segmentation architecture should in general be capable of segmenting streets AND cars.
Reduce the architecture of your model by applying Knowledge Distillation 1.

Cheers.


Module initialization error: An error occurred (403) when calling the HeadObject operation: Forbidden

Question:
Hello, everyone. I am trying to deploy main Text to speech trained model on AWS using lambda function.
There this error while loading my model from s3 bucket. I am also providing the code which of lambda function I have created and this error is occurring while testing the code in test configuration. Please help me resolve this issue.
try:
	import unzip_requirements
except ImportError:
	print('Unable to import');
import json
import urllib.parse
import boto3

print('Loading function')

s3 = boto3.client('s3')
import torch
import numpy as np
print(np.__version__);print(np.__file__)
#from scipy.io.wavfile import write
from os import path
print(torch.__version__)
print(torch.cuda.is_available())

my_bucket = 'modeltts'
orig_file = 'TTS_Nigerian'
dest_file = '/'

s3 = boto3.resource('s3')
s3.Bucket(my_bucket).download_file(orig_file, dest_file)

model_path= dest_file
tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')
tacotron2.load_state_dict(torch.load(model_path)['state_dict'])
tacotron2 = tacotron2.to('cuda')
tacotron2.eval()

def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    #bucket = event['Records'][0]['s3']['bucket']['name']
    #key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')
    #try:
        #response = s3.get_object(Bucket=bucket, Key=key)
        #print("CONTENT TYPE: " + response['ContentType'])
        #return response['ContentType']
    #except Exception as e:
        #print(e)
        #print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
        #raise e

    #s3 = boto3.resource('s3')
    #with open('TTS_Nigerian', wb) as tacotron2:
        #s3.Bucket(bucket).download_file(key, tacotron2)
    #tacotron2 = load_model()
    text = "Dear User, Your guider got closed unexpectedly.Would you like to continue from where you left off."

    utils = torch.hub.load('NVIDIA/DeepLearningExamples:torchhub', 'nvidia_tts_utils')
    sequence, lengths = utils.prepare_input_sequence([text])

    with torch.no_grad():

        mel_output_postnet, _, alignment = tacotron2.infer(sequence, lengths)
        audio = waveglow.infer(mel_output_postnet)

        #plot_data((mel_output_postnet.float().data.cpu().numpy()[0],
                    #alignment.float().data.cpu().numpy()[0].T))
    audio_numpy = audio[0].data.cpu().numpy()
    sampling_rate = 22050

    from IPython.display import Audio
    Audio(audio_numpy, rate=sampling_rate)

Context: @ptrblck please acknowledge


PyTorch-v1.7.1+cu110 - CUDA initialization error

Question:
I’m using p4d-24xlarge instance(NVIDIA A100) on AWS with CUDA/drivers showing installed correctly, but torch.cuda doesn’t load up. The instance has been setup using the step here.
Can anyone please tell me what might be causing this? Here are more info about my set up. Thanks in advance!
Error
conda install pytorch torchvision cudatoolkit=11.0 -c pytorch
python -c 'import torch; print(torch.cuda.is_available())'
/home_shared/valentyn/.conda/envs/cuda112/lib/python3.8/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at  /opt/conda/conda-bld/pytorch_1607370172916/work/c10/cuda/CUDAFunctions.cpp:100.)
  return torch._C._cuda_getDeviceCount() > 0
False

Installations
Ubuntu 18.04 LTS
$ uname -a
Linux worker-p4d-24xlarge-spot14 5.4.0-1037-aws #39~18.04.1-Ubuntu SMP Fri Jan 15 02:48:42 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

CUDA 11.2
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Jan_28_19:32:09_PST_2021
Cuda compilation tools, release 11.2, V11.2.142
Build cuda_11.2.r11.2/compiler.29558016_0

torch v1.7.1
>>> import torch; torch.__version__;
'1.7.1+cu110'

NVCC
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Jan_28_19:32:09_PST_2021
Cuda compilation tools, release 11.2, V11.2.142
Build cuda_11.2.r11.2/compiler.29558016_0

nvidia-smi
$ nvidia-smi
Thu Feb 25 16:25:54 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.39 Driver Version: 460.39 CUDA Version: 11.2 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 A100-SXM4-40GB Off | 00000000:10:1C.0 Off | 0 |
| N/A 44C P0 48W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 A100-SXM4-40GB Off | 00000000:10:1D.0 Off | 0 |
| N/A 40C P0 44W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 A100-SXM4-40GB Off | 00000000:20:1C.0 Off | 0 |
| N/A 42C P0 47W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 A100-SXM4-40GB Off | 00000000:20:1D.0 Off | 0 |
| N/A 40C P0 45W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 4 A100-SXM4-40GB Off | 00000000:90:1C.0 Off | 0 |
| N/A 42C P0 45W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 5 A100-SXM4-40GB Off | 00000000:90:1D.0 Off | 0 |
| N/A 40C P0 44W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 6 A100-SXM4-40GB Off | 00000000:A0:1C.0 Off | 0 |
| N/A 44C P0 46W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 7 A100-SXM4-40GB Off | 00000000:A0:1D.0 Off | 0 |
| N/A 43C P0 45W / 400W | 0MiB / 40536MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+

Data Center GPU Manager
$ dcgmi discovery -l
8 GPUs found.
+--------+----------------------------------------------------------------------+
| GPU ID | Device Information |
+--------+----------------------------------------------------------------------+
| 0 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:10:1C.0 |
| | Device UUID: GPU-ee16b34f-512a-99f3-3254-5d13653417eb |
+--------+----------------------------------------------------------------------+
| 1 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:10:1D.0 |
| | Device UUID: GPU-5b4c8387-ed2c-6d6b-5226-9a442a43556f |
+--------+----------------------------------------------------------------------+
| 2 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:20:1C.0 |
| | Device UUID: GPU-08ece2c6-fabb-7616-6970-40e5c4501cc0 |
+--------+----------------------------------------------------------------------+
| 3 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:20:1D.0 |
| | Device UUID: GPU-154ef609-926e-d809-89f6-46c643274c0e |
+--------+----------------------------------------------------------------------+
| 4 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:90:1C.0 |
| | Device UUID: GPU-4be0122e-75a0-6da1-bfcc-f6e814907f35 |
+--------+----------------------------------------------------------------------+
| 5 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:90:1D.0 |
| | Device UUID: GPU-c971220d-fba2-45c5-df64-f54fd8bf1a6c |
+--------+----------------------------------------------------------------------+
| 6 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:A0:1C.0 |
| | Device UUID: GPU-ddca000b-bdff-f303-233c-f92a46596a9a |
+--------+----------------------------------------------------------------------+
| 7 | Name: A100-SXM4-40GB |
| | PCI Bus ID: 00000000:A0:1D.0 |
| | Device UUID: GPU-0a31572a-40f9-39cc-09bd-783b9ee3dde1 |
+--------+----------------------------------------------------------------------+
0 NvSwitches found.
+-----------+
| Switch ID |
+-----------+
+-----------+

nvidia packages installed
$ dpkg -l | grep -i nvidia
ii cuda-nsight-compute-11-2 11.2.1-1 amd64 NVIDIA Nsight Compute
ii cuda-nsight-systems-11-2 11.2.1-1 amd64 NVIDIA Nsight Systems
ii cuda-nvtx-11-2 11.2.67-1 amd64 NVIDIA Tools Extension
ii datacenter-gpu-manager 1:2.1.4 amd64 NVIDIA® Datacenter GPU Management Tools
ii libnvidia-cfg1-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-460 460.39-0ubuntu0.18.04.1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVIDIA libcompute package
ii libnvidia-container-tools 1.3.3-1 amd64 NVIDIA container runtime library (command-line tools)
ii libnvidia-container1:amd64 1.3.3-1 amd64 NVIDIA container runtime library
ii libnvidia-decode-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-extra-460:amd64 460.39-0ubuntu0.18.04.1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ifr1-460:amd64 460.39-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii nsight-compute-2020.3.1 2020.3.1.3-1 amd64 NVIDIA Nsight Compute
ii nvidia-compute-utils-460 460.39-0ubuntu0.18.04.1 amd64 NVIDIA compute utilities
ii nvidia-container-runtime 3.4.2-1 amd64 NVIDIA container runtime
ii nvidia-container-toolkit 1.4.2-1 amd64 NVIDIA container runtime hook
ii nvidia-dkms-460 460.39-0ubuntu0.18.04.1 amd64 NVIDIA DKMS package
ii nvidia-docker2 2.5.0-1 all nvidia-docker CLI wrapper
ii nvidia-driver-460 460.39-0ubuntu0.18.04.1 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-460 460.39-0ubuntu0.18.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-460 460.39-0ubuntu0.18.04.1 amd64 NVIDIA kernel source package
ii nvidia-modprobe 460.32.03-0ubuntu1 amd64 Load the NVIDIA kernel driver and create device files
ii nvidia-prime 0.8.8.2 all Tools to enable NVIDIA's Prime
ii nvidia-settings 440.82-0ubuntu0.18.04.1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-460 460.39-0ubuntu0.18.04.1 amd64 NVIDIA driver support binaries
ii xserver-xorg-video-nvidia-460 460.39-0ubuntu0.18.04.1 amd64 NVIDIA binary Xorg driver

Thanks!

Context:
The error points towards a machine setup failure.
Could you try to compile the CUDA samples and run them?


Question:
Hello, I’m getting the following error message:
start java frontend failed: (<class ‘NotADirectoryError’>, NotADirectoryError(20, ‘Not a directory’), <traceback object at 0x7f972aa7ad20>), while trying to start the torchserve serve on AWS EC2, using the following command:
torchserve --start --model-store model_store --models densenet161=densenet161.mar
What am I doing wrong?
Thanks

Context: Based on the error message a directory doesn’t exist. I guess model_store might be the one, so make sure you’ve created it before executing torchserve.


Question:
Hi Pytorch Team,
I am converting one PyTorch Model(yoloact) trained on my custom dataset to ONNX, the conversion is completing successfully, but in the ONNX model one of the node is not getting connected to the graph.
I have attached the picture of the graph below.
Having said that, the pytorch model Inference is working fine, but not sure why one of the node is not getting connected to the graph in the onnx model.
Can you please assist me to solve this error? Would be a great help from yourside.
I am attaching the screenshot of the graph below:

Screenshot from 2021-05-18 12-17-131341×426 28.7 KB

Thanks,
Darshan

Context:
Looking forward to the Reply.
Regards,
Darshan


[HELP] Torch.onnx.export can not export onnx model

Question:
Hi , I’m trying to export my trained model that in .pt format to ONNX but after I execute the script the program output bunch of logging stuff like in the pic below and the converted onnx format seems not appear in my local disk  I don’t know what I got wrong here?

image1435×893 138 KB

Context: Does anyone know the answer?


Question:
Hello, I noticed that Pytorch 1.9 still uses cuDNN 8.0.5, same as 1.8 even though cuDNN 8.2.1 has been released. Is there a reason for this?
I read in the cuDNN release notes that there were some performance regressions for some legacy APIs since 8.0.5.

Context:
Before updating a cudnn version (or any other 3rd party library) we are usually running it for a couple of weeks in the nightlies to get visibility into potential issues.
In the 1.9.0 release we were working on fixing the library split issue 9, the missing cutlass/xmma kernels 3 etc.
While cudnn8.2.1 is already released, the risk was too high to fix the outstanding bugs and update cudnn without the visibility of the nightlies, which is why it’s delayed.


Initialization of register_buffer for loading model

Question:
Hello,
I am in a situation were I initialize and train a model using some datasets. I then save the model state dict. I would like to be able to load the model without having to load the data again. However because of the dimensions mismatch when I call load_state_dict I don’t seem to be able to do that.
example :
class Model(nn.Module):
    def __init__(
            self,
            layer_list,
            x_mean: Optional[np.array] = None,
            x_std: Optional[np.array] = None,
            **_
    ):
        super(Model, self).__init__()

        self.register_buffer('x_mean', None if x_mean is None else torch.FloatTensor(x_mean))
        self.register_buffer('x_std', None if x_std is None else torch.FloatTensor(x_std))
        self.sequential = get_sequential(layer_list)

After training the x_mean, x_std, … are filled with tensor of certain shapes. However when I load the model, I ‘don’t know’ the shape of those, and I would like to avoid to search in the data what are the shape of x_mean and x_std.
In this case I would like to load that model, just using the layer_list and the function load_state_dict
Thanks!

Context:
In your code snippet you are loading the state_dict into model, while I assume you want to load it into model_loaded, which will raise a RuntimeError, since no x buffer was registered:
import torch
from typing import Optional

class Model(torch.nn.Module):
    def __init__(self,x: Optional[list] = None):
        super(Model, self).__init__()
        self.register_buffer('x', None if x is None else torch.FloatTensor(x))

PATH = 'state_dict.pt'
x = [1,  2]

model = Model(x)
sd = model.state_dict()

model_loaded = Model()
model_loaded.load_state_dict(sd)
> RuntimeError: Error(s) in loading state_dict for Model:
	Unexpected key(s) in state_dict: "x".

The proper way would be to make sure to create this buffer, since you already know the shape from the loaded state_dict:
model_loaded = Model(sd['x'])
model_loaded.load_state_dict(sd)
> <All keys matched successfully>


How to handle different input image sizes?

Question:
Say I have an image model, how can I handle different image sizes?
Should I generate multiple onnx models for different sizes?
Do I do resizing or padding?
Thanks

Context:
The typical approach is to simply crop and resize to the trained model’s input size (so that the scale of the objects in terms of the number of pixels is close to what was seen at training time). You can generate different models for different size, but you still need to account for the scale issue; a model trained at 224x224 will start losing accuracy without finetuning around 400x400 resolution because of this scale mismatch problem.
[1906.06423] Fixing the train-test resolution discrepancy 3 is a good paper for more details on how to handle resizing


Question:
We are working on deploying PyTorch on Micro Controllers with different sensors.
We are looking for optimization of exported model with quantization, fusion and other memory reduction techniques before exporting the model to onnx.
I’ve searched PyTorch Docs, Tutorials and Mobile Demos without luck.
If you’ve worked, planning to work or have interest, please leave guidance, comment and other helpful resources.
Thank you!

Context:
Has there been any update on this?
I’m interested in trying this too.


Basic operations do not work in 1.1.0 with uWSGI+Flask

Question:
Came across the problem, that in pytorch 1.1.0 with uwsgi+flask on cpu even torch.cat does not work (everything just freezes without errors). I determined that problem is in uwsgi/flask, because in the same environment I was able to do the same operations without any issues. There was no such problem in previous versions of pytorch. As far as I understand, I am dealing with multiprocessing artifacts…
Nevertheless I found the solution and it was simple:
app = flask.Flask(__name__)
segmentator = None

@app.before_first_request
def load_segmentator():
    global segmentator
    segmentator = Segmentator()

where Segmentator is a class with pytorch’s nn.Module, which loads weights in __init__
Hope, it will help somebody.
P.S. If someone explains what is going on here, will be grateful.

Context: I’m facing the same issue. i tried a similar solution but still doesn’t seem to work. Any more information about this would be appreciated


Question:
I was trying to tailor-make the loss function to better reflect what I was trying to achieve.
As the results were not convincing I therefore tested an MSE loss self-coded versus the nn and F versions with :

criterion = nn.MSELoss() and loss1 = criterion1(outputs, targets)
loss2 = F.mse_loss(outputs, targets)
criterion3 = AdjMSELoss() and loss3 = criterion3(outputs, targets)
where :

class AdjMSELoss(nn.Module):
def init(self):
super(AdjMSELoss, self).init()
def forward(self, outputs, targets):
    outputs = torch.squeeze(outputs)
    loss = (outputs - targets)**2
    return torch.mean(loss)

As long as it test this with 2 tensors outside a backprop training, it gives the exact same results… BUT when I run a model with these losses for the backprop, it diverges between the AdjMSELoss custom function and the 2 nn.MSELoss and F.mse_loss (that gave identical results).
NB. All trials share the exact same seed and are reproducible
Could someone explain why (and advise on the best approach to create an efficient custom loss function) ?
THX in advance
The code for the custom loss function is given above.
The code for the training is as follows :

def train_model1D(model, criterion, optimizer, epochs, learning, verbose, train_dl):
model.train()
torch.set_grad_enabled(True)
total_step = len(train_dl)
curr_lr = learning
if str(criterion) != ‘CrossEntropyLoss()’:
criterion2 = AdjRegrLoss()
history = dict(train=[], train_regr=[])
else:
criterion2 = nn.CrossEntropyLoss()
accuracy = 0
history = dict(train=[], train_regr=[], acc=[])
best_model_wts = copy.deepcopy(model.state_dict())
best_loss = 1e64
for epoch in range(epochs):
train_losses = []
trregr_losses = []
y_preda = []
y_truea = []
for i, (inputs, targets) in enumerate(train_dl):
y_pred = model(inputs)
loss = criterion(y_pred, targets)
optimizer.zero_grad()
loss.backward()
optimizer.step()
train_losses.append(loss.item())
loss2 = criterion2(y_pred, targets)
trregr_losses.append(loss2.item())
if str(criterion) == ‘CrossEntropyLoss()’:
_, y_predb = torch.max(y_pred, 1)
y_predb = y_predb.cpu()
y_predb = y_predb.detach().numpy()
y_trueb = targets.cpu()
y_trueb = y_trueb.detach().numpy()
y_preda.extend(y_predb)
y_truea.extend(y_trueb)
train_loss = np.mean(train_losses)
trregr_loss = np.sum(trregr_losses)
history[‘train’].append(train_loss)
history[‘train_regr’].append(trregr_loss)
if str(criterion) == ‘CrossEntropyLoss()’:
accuracy = accuracy_score(y_preda, y_truea)
history[‘acc’].append(accuracy)
if trregr_loss < best_loss:
best_loss = trregr_loss
best_model_wts = copy.deepcopy(model.state_dict())
if ((verbose != 0) and (epoch == 0)):
print('Epoch 0  - Train Loss = ’ + str(round(train_loss, 6)) + ’ // Cumulative error training = ’ + str(round(trregr_loss, 6)))
if ((verbose != 0) and (((epoch+1) % verbose) ==0)):
print('Epoch ’ + str(epoch+1) + ’ - Train Loss = ’ + str(round(train_loss, 6)) + ’ // Cumulative error training = ’ + str(round(trregr_loss,6)))
torch.set_grad_enabled(False)
model.eval()
return model, history

You’ll notice that

I torch squeeze the output to get a tensor of size [batch, ] instead of [batch, 1]
I compute another loss function trregr_loss that is not relevant and not used for the back propagation. It computes the effective loss of prediction if yhat is of the opposite sign of ytrue.

class AdjRegrLoss(nn.Module):
def init(self):
super(AdjRegrLoss, self).init()
def forward(self, outputs, labels):
    outputs = torch.squeeze(outputs)
    loss = torch.abs(labels)
    adj = torch.mul(outputs, labels)
    adj[adj>0] = 0
    adj[adj<0] = 1
    loss = loss * adj
    return torch.sum(loss)

Context: Can you give the minimum codes to reproduce the error? And btw, if you want to implement the default init function of your custom class, that should be surrounded with two underscores on each side, like def __init__(self, *args)


Question:
When I try to export the “roi_align” operator in torchvision to onnx, I meet the following error.
Do you know how to fix it? Thank you!
RuntimeError: ONNX export failed on an operator with unrecognized namespace torchvision::roi_align. If you are trying to export a custom operator, make sure you registered it with the right domain and version. Otherwise, please report a bug.

Context: Based on the error message I guess you might need to register this custom operation as described here 51.


CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm`

Question:
Installed CUDA version:

NVIDIA-SMI 460.27.04
Driver Version: 460.27.04

CUDA Version: 11.2
OS:

Ubuntu 20.04

>>> python3 -m torch.utils.collect_env
Collecting environment information...
PyTorch version: 1.8.1+cu111
Is debug build: False
CUDA used to build PyTorch: 11.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti

Nvidia driver version: 460.27.04
cuDNN version: Probably one of the following:
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.3
[pip3] torch==1.8.1+cu111
[pip3] torchaudio==0.8.1
[pip3] torchvision==0.9.1+cu111
[conda] Could not collect

I am getting following error:
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Context: Try to install the 1.8.1 conda binaries or any nightly release, which should have fixed this issue.


Docker: Error 804: forward compatibility (…)

Question:
Hi all,
I fail to get a minimal, bleeding edge, CUDA-enabled pytorch container working. I followed the build instructions on pytorch’s README. On the same machine, nvidia’s pytorch image 3 works just fine, launched with the same docker run arguments. The only difference I could spot was CUDA 11.2 in nvidia’s image vs CUDA 11.3 in mine. Is this known not to work? I am rebuilding with CUDA 11.2 right now but it takes a while, so I figured I’d ask here in the meantime in case I am missing something obvious and someone sees this.
This is what I get (inside the container):
root@42afc3dc16b0:/# nvidia-smi
Fri May 28 15:00:14 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.3     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GT 710      On   | 00000000:09:00.0 N/A |                  N/A |
| 50%   43C    P8    N/A /  N/A |    222MiB /   973MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1080    On   | 00000000:0A:00.0 Off |                  N/A |
|  0%   29C    P8     8W / 210W |      6MiB /  8119MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
root@42afc3dc16b0:/# python3 -c "import torch;print(torch.cuda.is_available())"
/opt/miniconda/lib/python3.9/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 804: forward compatibility was attempted on non supported HW (Triggered internally at  ../c10/cuda/CUDAFunctions.cpp:115.)
  return torch._C._cuda_getDeviceCount() > 0
False

This is my Dockerfile:
FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 AS condainstall

COPY Miniconda3-py39_4.9.2-Linux-x86_64.sh /root/
RUN sh /root/Miniconda3-py39_4.9.2-Linux-x86_64.sh -b -p /opt/miniconda && \
    /opt/miniconda/bin/conda install -y\
        astunparse \
        numpy\
        ninja\
        pyyaml\
        mkl\
        mkl-include\
        setuptools\
        cmake\
        cffi\
        typing_extensions\
        future\
        six\
        requests\
        dataclasses && \
    /opt/miniconda/bin/conda clean --all

FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04 AS torchbuild

COPY --from=condainstall /opt/miniconda /opt/miniconda

# cloned from github
COPY pytorch /root/pytorch

WORKDIR /root/pytorch

RUN apt-get update && apt install gcc g++ -y

RUN eval "$(/opt/miniconda/bin/conda shell.bash hook)" && \
    export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} && \
    python3 setup.py install

FROM nvidia/cuda:11.3.0-cudnn8-devel-ubuntu20.04

COPY --from=torchbuild /opt/miniconda /opt/miniconda

ENV PATH /opt/miniconda/bin:$PATH

ENTRYPOINT ["python3", "-c", "import torch;print(torch.cuda.is_available())"]

Context:
It turns out using CUDA 11.2 instead of 11.3 solved my problem.
Thanks for your reply @ptrblck. Updating the drivers on the bare metal might work too, but I am happy to just let debian’s APT handle the drivers on the bare metal node for now.


Question:
Hello, I’m trying create my own base_handler by Documentation, but my skills are low to understand how correct write for my task. (I’m Junior DS  )
Very need any help!
I have wrote script which just processes the data, feeds it to the Binary Classification model and outputs the result
I a lot tried describe correctly by example serve/base_handler.py at master · pytorch/serve · GitHub 3 and other examples but I can’t understand if I correct write all code
curl:
curl -F data=@test.csv http://127.0.0.1:8080/predictions/binclaff
{
  "code": 503,
  "type": "InternalServerException",
  "message": "Prediction failed"
}

Log:
2021-05-24 15:01:30,963 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Invoking custom service failed.
2021-05-24 15:01:30,963 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - Traceback (most recent call last):
2021-05-24 15:01:30,963 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/ts/service.py", line 100, in predict
2021-05-24 15:01:30,963 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     ret = self._entry_point(input_batch, self.context)
2021-05-24 15:01:30,963 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/tmp/models/705a8b53edde46d384eaf6b6f46d6802/my_handler.py", line 191, in handle
2021-05-24 15:01:30,963 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     data = _service.preprocess(data)
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/tmp/models/705a8b53edde46d384eaf6b6f46d6802/my_handler.py", line 89, in preprocess
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     df = read_csv(filename, dtype={'SessionId': uint64})
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 610, in read_csv
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     return _read(filepath_or_buffer, kwds)
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 462, in _read
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     parser = TextFileReader(filepath_or_buffer, **kwds)
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 819, in __init__
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self._engine = self._make_engine(self.engine)
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1050, in _make_engine
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1 org.pytorch.serve.wlm.WorkerThread - Backend response time: 4
2021-05-24 15:01:30,964 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     return mapping[engine](self.f, **self.options)  # type: ignore[call-arg]
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1867, in __init__
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self._open_handles(src, kwds)
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/parsers.py", line 1362, in _open_handles
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1 ACCESS_LOG - /127.0.0.1:42598 "POST /predictions/binclaff HTTP/1.1" 503 32
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1 TS_METRICS - Requests5XX.Count:1|#Level:Host|#hostname:alexey-GB-BKi5-H-A-7200,timestamp:null
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     self.handles = get_handle(
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/common.py", line 558, in get_handle
2021-05-24 15:01:30,965 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     ioargs = _get_filepath_or_buffer(
2021-05-24 15:01:30,966 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -   File "/home/alexey/.local/lib/python3.8/site-packages/pandas/io/common.py", line 371, in _get_filepath_or_buffer
2021-05-24 15:01:30,966 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle -     raise ValueError(msg)
2021-05-24 15:01:30,966 [INFO ] W-9000-binclaff_0.1-stdout org.pytorch.serve.wlm.WorkerLifeCycle - ValueError: Invalid file path or buffer object type: <class 'list'>
2021-05-24 15:01:30,965 [DEBUG] W-9000-binclaff_0.1 org.pytorch.serve.job.Job - Waiting time ns: 123775, Inference time ns: 5403468

My Script
import pathlib

from numpy import uint64

from pandas import read_csv
from pandas import DataFrame
from pandas import SparseDtype
from pandas import concat

from sys import argv
import joblib
import torch
from torch.utils.data import Dataset, DataLoader
from mlxtend.preprocessing import TransactionEncoder

script, filename = argv

absolutePAth = pathlib.Path(__file__).parent.absolute()
model = "%s/jit_model_best.pt" % absolutePAth
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

class Prepocessing(DataLoader):
    # load the dataset
    def __init__(self, path):
        # load the csv file as a dataframe
        self.df = read_csv(path, dtype={'SessionId': uint64})

    def data_tranform(self):
        self.df = self.df.drop({'IdentityId', 'SessionId', 'IsBot', 'ClientHash', 'ClientIP', 'UserAgent', 'AccountId',
                                'is_bot'},
                               axis=1, errors='ignore')

        self.df.external_referer_domains = self.df.external_referer_domains.str.replace(r'\]\[\'', '', regex=True)
        self.df['external_referer_domains'] = self.df.external_referer_domains.str.findall(
            '(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-&?=%.]+')
        self.df['count_external_referer_domains'] = self.df['external_referer_domains'].apply(lambda x: len(x))

        # нормальный массив из значений ивента и роутов
        # подсчет колличество используеммых 1 пользователем ивентов и маршрутов
        self.df['events'] = self.df.events.str.findall(r"\d+")
        self.df['unique_routes'] = self.df.unique_routes.str.findall(r"\d+")
        self.df['count_events'] = self.df['events'].apply(lambda x: len(x))
        self.df['count_unique_routes'] = self.df['unique_routes'].apply(lambda x: len(x))

        self.df['101'] = self.df.unique_routes.apply(lambda x: int('101' in x))
        self.df['282'] = self.df.unique_routes.apply(lambda x: int('282' in x))
        self.df['281'] = self.df.unique_routes.apply(lambda x: int('281' in x))
        self.df['266'] = self.df.unique_routes.apply(lambda x: int('266' in x))
        self.df['271'] = self.df.unique_routes.apply(lambda x: int('271' in x))
        self.df['279'] = self.df.unique_routes.apply(lambda x: int('279' in x))
        self.df['1'] = self.df.unique_routes.apply(lambda x: int('1' in x))
        self.df['5'] = self.df.unique_routes.apply(lambda x: int('5' in x))
        self.df['203'] = self.df.unique_routes.apply(lambda x: int('203' in x))

        self.df = self.df.fillna(0)

        # transaction encoder for events
        te = TransactionEncoder()
        te.__dict__['columns_mapping_'] = {str(i): i for i in list(range(48))}
        te_ary = te.fit(self.df.events).transform(self.df.events, sparse=True)

        if te_ary.shape[1] < 48:
            te.__dict__['columns_mapping_'] = {str(i): i for i in list(range(te_ary.shape[1]))}
            self.df_events = DataFrame.sparse.from_spmatrix(te_ary, columns=te.__dict__['columns_mapping_'].keys())
            a = 48 - te_ary.shape[1]
            b = 48 - a
            for i in list(range(b, 48)):
                self.df_events[i] = int(0)
                self.df_events = self.df_events.astype(SparseDtype(int))

        else:
            self.df_events = DataFrame.sparse.from_spmatrix(te_ary, columns=te.__dict__['columns_mapping_'].keys())
            self.df_events = self.df_events.astype(SparseDtype(int))

        self.df_events = self.df_events.add_prefix('event_')

        # drop events and routes bec we have spartse matrix
        self.df = self.df.drop({'events', 'unique_routes', 'external_referer_domains'}, axis=1)
        self.df = concat([self.df, self.df_events], axis=1, join="inner")

        # load StandardScaler
        scaler = joblib.load("%s/std_scaler_new.bin" % absolutePAth)
        self.df = scaler.transform(self.df.values)

        test_loader = DataLoader(dataset=torch.FloatTensor(self.df), batch_size=1)

        return test_loader

    def load_data(path):
        df = read_csv(path, dtype={'SessionId': uint64})
        return df

test_loader = Prepocessing(filename).data_tranform()

the_model = torch.jit.load(model, map_location=device)
the_model.eval()

y_pred_list = []
with torch.no_grad():
    for X_batch in test_loader:
        X_batch = X_batch.cpu()
        y_test_pred = the_model.cpu()(X_batch)
        y_test_pred = torch.sigmoid(y_test_pred)
        y_pred_tag = torch.round(y_test_pred)
        y_pred_list.append(y_pred_tag.cpu().numpy())

y_pred_list = [a.squeeze().tolist() for a in y_pred_list]

df = Prepocessing.load_data(filename)
df = DataFrame(
    {"SessionId": df.SessionId,
     "IsBot": y_pred_list})

My own fail handler
import logging
import os
import io
import re
import abc
from sys import argv
import joblib
import telegram
import csv
from numpy import uint64
from pandas import read_csv
from pandas import DataFrame
from pandas import SparseDtype
from pandas import concat
from mlxtend.preprocessing import TransactionEncoder
from ts.torch_handler.base_handler import BaseHandler

import torch
from torch.utils.data import Dataset, DataLoader

logger = logging.getLogger(__name__)

def tlog2(message):
    token = "1809630354:AAGAYquyq6Aev5obJJ_7W8H8xCqX36AVX2o"
    chat_id = "-557190848"
    bot = telegram.Bot(token=token)
    bot.sendMessage(chat_id=chat_id, text=message)

class MyHandler(BaseHandler):
    """
    Base default handler to load torchscript or eager mode [state_dict] models
    Also, provides handle method per torch serve custom model specification
    """

    def __init__(self):
        super(MyHandler, self).__init__()
        self.model = None
        self.mapping = None
        self.device = None
        self.initialized = False
        self.context = None
        self.manifest = None
        self.map_location = None
        self.explain = False
        self.target = 0

    def initialize(self, context):
        """Initialize function loads the model.pt file and initialized the model object.
	   First try to load torchscript else load eager mode state_dict based model.
        Args:
            context (context): It is a JSON Object containing information
            pertaining to the model artifacts parameters.
        Raises:
            RuntimeError: Raises the Runtime error when the model.py is missing
        """
        properties = context.system_properties
        # на чем будет работать GPU or CPU
        self.map_location = "cuda" if torch.cuda.is_available() and properties.get("gpu_id") is not None else "cpu"
        self.device = torch.device(
            self.map_location + ":" + str(properties.get("gpu_id"))
            if torch.cuda.is_available() and properties.get("gpu_id") is not None
            else self.map_location
        )
        self.manifest = context.manifest

        model_dir = properties.get("model_dir")
        model_pt_path = None
        # определяем где находиться файл модель
        if "serializedFile" in self.manifest["model"]:
            serialized_file = self.manifest["model"]["serializedFile"]
            model_pt_path = os.path.join(model_dir, serialized_file)

        self.model = torch.jit.load(model_pt_path, map_location=self.device)
        self.model.eval()

        logger.debug('Model file %s loaded successfully', model_pt_path)

        self.initialized = True

        tlog2('initialize прошла успешно')

    def preprocess(self, filename):
        """
        Вставляем наш файл csv, который в дальнейшем открывается через Pandas и делает все обработки
        :param filename:
        :return:
        На выходе получаем загружанные данные в PyTorch DataLoader
        """
        df = read_csv(filename, dtype={'SessionId': uint64})
        tlog2('файл открылся успешно прошла успешно')
        df = df.drop({'IdentityId', 'SessionId', 'IsBot', 'ClientHash', 'ClientIP', 'UserAgent', 'AccountId',
                                'is_bot', 'is_robot'},
                               axis=1, errors='ignore')

        df.external_referer_domains = df.external_referer_domains.str.replace(r'\]\[\'', '', regex=True)
        df['external_referer_domains'] = df.external_referer_domains.str.findall(
            '(?:(?:https?|ftp):\/\/)?[\w/\-?=%.]+\.[\w/\-&?=%.]+')
        df['count_external_referer_domains'] = df['external_referer_domains'].apply(lambda x: len(x))

        # нормальный массив из значений ивента и роутов
        # подсчет колличество используеммых 1 пользователем ивентов и маршрутов
        df['events'] = df.events.str.findall(r"\d+")
        df['unique_routes'] = df.unique_routes.str.findall(r"\d+")
        df['count_events'] = df['events'].apply(lambda x: len(x))
        df['count_unique_routes'] = df['unique_routes'].apply(lambda x: len(x))

        df['101'] = df.unique_routes.apply(lambda x: int('101' in x))
        df['282'] = df.unique_routes.apply(lambda x: int('282' in x))
        df['281'] = df.unique_routes.apply(lambda x: int('281' in x))
        df['266'] = df.unique_routes.apply(lambda x: int('266' in x))
        df['271'] = df.unique_routes.apply(lambda x: int('271' in x))
        df['279'] = df.unique_routes.apply(lambda x: int('279' in x))
        df['1'] = df.unique_routes.apply(lambda x: int('1' in x))
        df['5'] = df.unique_routes.apply(lambda x: int('5' in x))
        df['203'] = df.unique_routes.apply(lambda x: int('203' in x))
        df['275'] = df.unique_routes.apply(lambda x: int('275' in x))

        df = df.fillna(0)

        # transaction encoder for events
        te = TransactionEncoder()
        te.__dict__['columns_mapping_'] = {str(i): i for i in list(range(48))}
        te_ary = te.fit(df.events).transform(df.events, sparse=True)

        if te_ary.shape[1] < 48:
            te.__dict__['columns_mapping_'] = {str(i): i for i in list(range(te_ary.shape[1]))}
            df_events = DataFrame.sparse.from_spmatrix(te_ary, columns=te.__dict__['columns_mapping_'].keys())
            a = 48 - te_ary.shape[1]
            b = 48 - a
            for i in list(range(b, 48)):
                df_events[i] = int(0)
                df_events = df_events.astype(SparseDtype(int))

        else:
            df_events = DataFrame.sparse.from_spmatrix(te_ary, columns=te.__dict__['columns_mapping_'].keys())
            df_events = df_events.astype(SparseDtype(int))

        df_events = df_events.add_prefix('event_')

        # drop events and routes bec we have spartse matrix
        df = df.drop({'events', 'unique_routes', 'external_referer_domains'}, axis=1)
        df = concat([df, df_events], axis=1, join="inner")

        # load StandardScaler
        scaler = joblib.load("scaler.bin")
        df = scaler.transform(df.values)

        df = torch.FloatTensor(df)
        test_loader = DataLoader(dataset=df, batch_size=1)

        return test_loader

    def inference(self, data):

        y_pred_list = []
        with torch.no_grad():
            for X_batch in data:
                X_batch = X_batch.cpu()
                y_test_pred = self.model.cpu()(X_batch)
                y_test_pred = torch.sigmoid(y_test_pred)
                y_pred_tag = torch.round(y_test_pred)
                y_pred_list.append(y_pred_tag.cpu().numpy())
                y_pred_list = [a.squeeze().tolist() for a in y_pred_list]

        return y_pred_list

    def postprocess(self, preds):
        # TODO: Add any needed post-processing of the model predictions here

        res = []
        # pres has size [BATCH_SIZE, 1]
        # convert it to list
        preds = preds.cpu().tolist()
        for pred in preds:
            label = self.mapping[str(pred)][1]
            res.append({'label': label, 'index': pred})
        return res

_service = MyHandler()

def handle(data, context):
    if not _service.initialized:
        _service.initialize(context)

    if data is None:
        return None

    data = _service.preprocess(data)
    data = _service.inference(data)
    data = _service.postprocess(data)

    return data

my model mar in DropBox: Dropbox - binclaff.mar - Simplify your life
I will be very grateful if someone can help me correctly describe my script in the handler and show my mistakes.

Context:
I fixed this issue. Solution for send request with csv file:
 df = read_csv(StringIO(path[0].get("body").decode()), dtype={'SessionId': uint64, 'IdentityId': uint64})


How to convert checkpoint file (.ckpt) to state dict file (.pth)?

Question:
Hi! New PyTorch user here
I am trained my model using Pytorch Lighting and ModelCheckpoint with parameter save_top_k=1, so only the best checkpoint is saved. . After the training is finished I saved the model as usual with torch.save(model.state_dict()).
Now I want to deploy my model for inference. My epoch=42.ckpt file contains a model with better performance than the final model, so I want to use this checkpoint file. But the checkpoint file is three times larger than the normal model file (.pth).
How do I convert a .ckpt file to a .pth file?

Context: CC @williamFalcon and @justusschock for Lightning questions


PyTorch’s matmul triggers different kernels to cublasSgemm

Question:
I’m trying to write a batched-gemm operator where each input has different shapes, following is a script I used for profiling and debugging:
import torch
import myop
import time

m = torch.tensor([100, 128, 20, 284, 89, 10, 82, 92, 10, 20, 30, 100, 128, 20, 284, 89, 10, 82, 92, 10, 20, 30,
                  100, 128, 20, 284, 89, 10, 82, 92, 10, 20, 30, 100, 128, 20, 284, 89, 10, 82, 92, 10, 20, 30])
k = torch.tensor([48, 256, 33, 92, 17, 520,  48, 256, 33, 92, 17, 520, 48, 256, 33, 92, 17, 520, 33, 92, 17, 520,
                  48, 256, 33, 92, 17, 520,  48, 256, 33, 92, 17, 520, 48, 256, 33, 92, 17, 520, 33, 92, 17, 520])
n = torch.tensor([200, 30, 17, 30, 200, 30, 17, 30, 200, 30, 17, 30, 200, 30, 17, 30, 200, 30, 17, 30, 200, 30,
                  200, 30, 17, 30, 200, 30, 17, 30, 200, 30, 17, 30, 200, 30, 17, 30, 200, 30, 17, 30, 200, 30])

A_arr = []
B_arr = []
C_arr = []

for mi, ni, ki in zip(m.tolist(), n.tolist(), k.tolist()):
    A_arr.append(torch.rand(mi, ki).to(0))
    B_arr.append(torch.rand(ki, ni).to(0))
    C_arr.append(torch.rand(mi, ni).to(0))

A = torch.cat([_.view(-1) for _ in A_arr], 0)
B = torch.cat([_.view(-1) for _ in B_arr], 0)
C = torch.cat([_.view(-1) for _ in C_arr], 0)

torch.cuda.synchronize()
tic = time.time()
for _ in range(100):
    for i in range(len(m)):
        Ai = A_arr[i]
        Bi = B_arr[i]
        Ci = Ai @ Bi

torch.cuda.synchronize()
toc = time.time()
print(toc - tic)

tic = time.time()
for _ in range(100):
    myop.multiple_gemm(A, B, C, m, n, k, False, False)

torch.cuda.synchronize()
toc = time.time()
print(toc - tic)

where myop is a C++ extension I wrote which basically calls cublasSgemm (non-transpose for both operands) operator for each (a, b, c) in the batch sequentially and only uses a single stream.
However, I found that myop’s execution time is longer than sequentially apply torch.matmul in Python. To investigate the reason I use nsight-system to get more insights:

image1346×91 22.9 KB

I found that PyTorch only triggers sgemm_32x32x32_NN_vec kernel while myop trigger multiple kinds of kernels including maxwell_sgemm_128_64_nn and sgemm_32x32x32_NN_vec, and I can confirm that both ways produce exactly the same result. I was wondering that what’s PyTorch behavior for GEMM on GPU, has it enabled some environment variable that might force cublas to not select kernel in the default way, etc?

Context: No, cublas env variables are not set, so I assume your manual implementation might just call into more cublas operations. Are these operations always called in your implementation?


PyTorch model to Onnx to TensorRT engine = no speed up for inference?

Question:
Hey everyone,
I’m working with a Jetson Nano device, TRT 6 (the latest version that can be used on the Nano), PyTorch 1.2.0 (compatible with TRT6), and Torchvision 0.4.0 (compatible with PyTorch 1.2.0).
I have a Torchvision Mobilenetv2 model I exported to Onnx with the built-in function:
    torch.onnx.export(pt_model, dummy_input, out_path, verbose=True)

I then built a TensorRt engine with this Onnx model:
with trt.Builder(TRT_LOGGER) as builder, \
      builder.create_network(*EXPLICIT_BATCH) as network, \
      trt.OnnxParser(network, TRT_LOGGER) as parser:

      builder.max_workspace_size = 1 << 28
      builder.max_batch_size = 1
      builder.fp16_mode = True
      # ...
      engine = builder.build_cuda_engine(network)

Then I run inference on this new engine on my Jetson Nano device and can get a latency of about 0.045 seconds (22.2 fps). Running inference on the PyTorch version of this model also has almost the exact same latency of 0.045 seconds.
I also tried to change the mode to INT8 mode when building the TensorRT engine and get the error: Builder failed while configuring INT8 mode.
Anyone have experience with optimizing Torch models with TensorRT? Am I missing something fundamental when building the TensorRT engine or should I expect a speed-up?
Thanks.

Context:
I haven’t tried to export Mobilenet yet, but could see a speedup for FasterRCNN.
How are you currently measuring the the throughput?


Question: Why are there two identical libraries inside LibTorch: libdnnl and libmlkdnn?

Context:
I would assume one of them is a symbolic link, but you are right. It seems both are shipped and should be identical, if I’m not mistaken:
ll libtorch/lib
[...]
185727360 Mar 23 21:02 libdnnl.a
185727360 Mar 23 21:02 libmkldnn.a
[...]

Could you create an issue on GitHub and link to this thread, please?


How to compile pytorch in an executable?

Question:
Hi, Im trying to compile a game using a pytorch model for part of the game. I want it in an executable format. For some context, I am doing this using cx_freeze on windows
But im getting a compilation error as shown in the image below. (Sorry, was unable to extract the text efficiently)
photo_2020-07-15_00-22-46414×965 91.6 KB
I dug around the errors a bit, and saw that the issue was related to how _VF is importing torch from within torch itself? I am unsure how to amend this though.
Any suggestions as to how I can troubleshoot this? The only thing i need the model to be able to do in compilation after training the weights before all this is to feedforward an image/stack of images and come to a predicted action.
Any advise or suggestion is much appreciated! Thanks!

Context:
Same problem here.
Cannot freeze torch application appropriately, due to the import issue of ‘_VF’