Windows uses twice as much VRAM than Linux for same code

Question:
I am running Tacotron2 on Windows with Conda environment with an RTX 3090.
I can get to run but in Windows, it seems like it uses twice as much VRAM as compared to my Linux counter parts.
I’ve tried using python 3.6 and 3.7. installing pytorch through pip and conda. using cudatoolkit.
CudaToolkits pre-v11 tell me its not sm_86 compatible and freeze.
I didn’t always have this issue but I’ve gone back to my conda env backups, and those don’t help.
Is there any scripts that can tell me which cuda version that its using? I removed all the cuda paths but somehow my conda env still knew where to find it and run.
My colab graphics card is 16GB and can do a batch of 48. While my windows graphics card is 24GB and can only do 32 batch size.
Colab Link:

      colab.research.google.com

Google Colaboratory

Context:
I installed cudnn 8.1.0 but when I run this code and I get: (‘11.0’, 8004)
def get_cuda_version():
    import torch
    cuda = (torch.version.cuda)
    cudnn = torch.backends.cudnn.version()
    return cuda, cudnn

print(get_cuda_version())


Initial Pytorch setup and another few questions

Question:
Hi everyone .
I’m new to Pytorch and I’m trying to figure out a few things right now .
I have the following components :
   ...:
`__Python VERSION: 3.6.5 |Anaconda, Inc.| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)]
__pyTorch VERSION: 1.2.0+cu92
__CUDA VERSION
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Tue_Jun_12_23:08:12_Central_Daylight_Time_2018
Cuda compilation tools, release 9.2, V9.2.148
__CUDNN VERSION: 7201
__Number CUDA Devices: 1
__Devices
Active CUDA Device: GPU 0
Available devices  1
Current cuda device  0``

I’m using the following hardware and software :
Z440 G4
16GB RAM
NVIDIA QUADRO RTX 4000 (CUDA supported)
Windows server 2012r2
Anaconda 3.5.2 - Python 3.6.5 ( I need that specific version) .

After the installation of all the components I needed I searched for a training example just to make sure Pytorch is working . I executed the following :

`import torch
 import torch.optim as optim

 model = torch.nn.Linear(5, 2)

  Initialize optimizer
 optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

 print("Model's state_dict:")
 for param_tensor in model.state_dict():
     print(param_tensor, "\t", model.state_dict()[param_tensor].size())

 print("Model weight:")
 print(model.weight)

 print("Model bias:")
 print(model.bias)

 print("---")
 print("Optimizer's state_dict:")
 for var_name in optimizer.state_dict():
     print(var_name, "\t", optimizer.state_dict()[var_name])``

And got the following output
Model's state_dict:
weight   torch.Size([2, 5])
bias     torch.Size([2])
Model weight:
Parameter containing:
tensor([[-0.1759, -0.2537,  0.1401,  0.2342, -0.3865],
        [-0.2142,  0.2763, -0.2919,  0.2003, -0.3242]], requires_grad=True)
Model bias:
Parameter containing:
tensor([-0.0585,  0.2511], requires_grad=True)
---
Optimizer's state_dict:
state    {}
param_groups     [{'lr': 0.001, 'momentum': 0.9, 'dampening': 0, 'weight_decay': 0, 'nesterov': False, 'params': [935863832152, 9358638
31360]}]

Next I have saved it by using :
torch.save(model.state_dict(), "temp.pt")

So far it seems OK to me . Anything else I can / Should check in addition ?
2 . How can I check in real time that my GPU is working ? I want to make sure PYTORCH is using the GPU and not the CPU . Anything I can check ? Saved log file maybe ?
Appreciate your help .
Please let me know if any information is needed .

Context:
NoamAS:

How can I check in real time that my GPU is working ?

Push your model to the device and execute a training step:
device = 'cuda'
model.to(device)
data = torch.randn(1, 5, device=device)
out = model(data)
out.mean().backward()

If this code runs fine then your GPU is used.
For more testing, you could execute some examples 2.


Should Pytorch install in develop mode using 100% CPU?

Question:
Hello, I built Pytorch from source and installed in develop mode for making open-source contributions. However, I noticed that when I have to update my local repository to keep up-to-date with the main repository, I have to re-install Pytorch in develop mode as stated in the contributing doc by running python setup.py develop (Step 3).
Running this script takes around 40 minutes on a brand new Intel i7-10700K (8 core with hyperthreading) and uses 100% of the CPU. I am wondering if this is normal or is there a way I can reduce CPU usage while running this script? Thank you.

Context: You could set the MAX_JOBS env variable to a specific number to reduce the CPU workload.


I cannot use the pytorch that was built successfully from source: (DLL) initialization routine failed. Error loading caffe2_detectron_ops_gpu.dll

Question: I cannot post more than two links here as a beginner, and which was definitely needed. That is why the whole issue is at https://github.com/pytorch/pytorch/issues/43210 93 but shall be answered here.

Context:
After so many tries, the following has worked for me.
I had to set ninja off. Ninja is in order to speed up the process, too bad that I cannot use it. Without ninja, it ran through the whole night for about 9.5 hours. I also needed to download the source code of MKL, and then, together with the mkl installation and other tricks (which are all documented above), it works:

(myenv) C:\WINDOWS\system32>cd C:\Users\Admin\Downloads\Pytorch\pytorch
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “CMAKE_INCLUDE_PATH=C:\Users\Admin\Downloads\Pytorch\mkl\include”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “LIB=C:\Users\Admin\Downloads\Pytorch\mkl\lib;%LIB%”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_NINJA=OFF
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set CMAKE_GENERATOR=Visual Studio 16 2019
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set USE_MKLDNN=ON
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>set “CUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.27.29110\bin\Hostx64\x64\cl.exe”
(myenv) C:\Users\Admin\Downloads\Pytorch\pytorch>python setup.py install --cmake

This has run through, yet the spyder was corrupted by the cuda install:
(myenv) C:\WINDOWS\system32>spyder
_cffi_ext.c
C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\cffi\__pycache__\_cffi_ext.c(268): fatal error C1083: Datei (Include) kann nicht geöffnet werden: "zmq.h": No such file or directory
Traceback (most recent call last):
  File "C:\Users\Admin\anaconda3\Scripts\spyder-script.py", line 6, in <module>
    from spyder.app.start import main
  File "C:\Users\Admin\anaconda3\lib\site-packages\spyder\app\start.py", line 22, in <module>
    import zmq
  File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\__init__.py", line 50, in <module>
    from zmq import backend
  File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\__init__.py", line 40, in <module>
    reraise(*exc_info)
  File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\utils\sixcerpt.py", line 34, in reraise
    raise value
  File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\__init__.py", line 27, in <module>
    _ns = select_backend(first)
  File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\select.py", line 28, in select_backend
    mod = __import__(name, fromlist=public_api)
  File "C:\Users\Admin\anaconda3\lib\site-packages\zmq\backend\cython\__init__.py", line 6, in <module>
    from . import (constants, error, message, context,
ImportError: DLL load failed while importing error: Das angegebene Modul wurde nicht gefunden.

Installing it over the existing installation again:

(myenv) C:\WINDOWS\system32>conda install spyder

Opening spyder:
(myenv) C:\WINDOWS\system32>spyder
Test according to this 3:
import torch

torch.__version__
Out[3]: '1.8.0a0+2ab74a4'

torch.cuda.current_device()
Out[4]: 0

torch.cuda.device(0)
Out[5]: <torch.cuda.device at 0x24e6b98a400>

torch.cuda.device_count()
Out[6]: 1

torch.cuda.get_device_name(0)
Out[7]: 'GeForce GT 710'

torch.cuda.is_available()
Out[8]: True

Finally
++ A compressed summary of all steps can be found on Stack Exchange at https://superuser.com/questions/1596115/how-to-install-pytorch-with-cuda-enabled-for-a-deprecated-cuda-cc-3-5-of-an-old 40 ++


Question:
E:\Python35>python work\train.py
After executing the above code, there is the following bug.
My system is windows, and My cifar data is in the file folder, E:\Python35\work\data
The error is as follows: Thank you in advance.
Traceback (most recent call last):
File “work\train.py”, line 39, in
trainset = torchvision.datasets.CIFAR10(root=‘E:\Python35\work\data’, train=True, download=False, transform=transform_train) #训练数据集
File “E:\Python35\lib\site-packages\torchvision\datasets\cifar.py”, line 62, in init
’ You can use download=True to download it’)
RuntimeError: Dataset not found or corrupted. You can use download=True to download it
Exception ignored in: <function WeakValueDictionary.init..remove at 0x0000027BF28DAF28>
Traceback (most recent call last):
File “E:\Python35\lib\weakref.py”, line 117, in remove
TypeError: ‘NoneType’ object is not callable

Context:
You might have to use \\ or / in the path, since \ might be understood as an escape sequence.
If the error is still raised, you could try to download it again via download=True.


Does Pytorch 1.7 support CUDA Toolkit 11.0 or CUDA Toolkit 11.0 Update1?

Question:
In this web page, there are two versions,  CUDA Toolkit 11.0 and CUDA Toolkit 11.0 Update1 . CUDA Toolkit Archive | NVIDIA Developer 2
I am confused. If I want to install Pytorch version with CUDA, Which CUDA should I choose?

Context:
Double post from here 17 with a potential answer.
Cross-posting:

you don’t need a local CUDA toolkit installation if you don’t plan on building PyTorch from source or any CUDA extension, just the NVIDIA driver.
the binaries use CUDA11.0.3, so 11.0 Update 1. If you plan on building custom CUDA extensions, you should install the same local toolkit


PyTorch 1.7.1 CUDA 11.1 Windows build failed

Question:
Hi
I’m trying to build PyTorch 1.7.1 from source using CUDA 11.1 on Windows (build worked fine on Ubuntu 18.04, now needs the Windows version)
I’m following this thread: PyTorch build from source on Windows - #17 by mmisiur 4 and I keep getting these errors:
..\aten\src\ATen/core/dispatch/Dispatcher.h(368): error C2039: 'boxArgs': is not a member of 'c10::impl::BoxedKernelWrapper<Return (at::Tensor &,const at::Tensor &,c10::optional<c10::MemoryFormat>),void>'
        with
        [
            Return=at::Tensor &
        ]
..\aten\src\ATen/core/dispatch/Dispatcher.h(368): note: see declaration of 'c10::impl::BoxedKernelWrapper<Return (at::Tensor &,const at::Tensor &,c10::optional<c10::MemoryFormat>),void>'
        with
        [
            Return=at::Tensor &
        ]
..\aten\src\ATen/core/boxing/impl/boxing.h(100): error C2070: 'FuncType': illegal sizeof operand
        with
        [
            FuncType=at::Tensor (const at::Tensor &,c10::optional<c10::MemoryFormat>)
        ]

CMake summary output (not enough room to post entire output, will post later if needed):
--
-- ******** Summary ********
-- General:
--   CMake version         : 3.19.2
--   CMake command         : C:/Users/cbarak/.conda/envs/pytorch_1.7.1/Library/bin/cmake.exe
--   System                : Windows
--   C++ compiler          : C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/cl.exe
--   C++ compiler id       : MSVC
--   C++ compiler version  : 19.16.27040.0
--   BLAS                  : MKL
--   CXX flags             : /DWIN32 /D_WINDOWS /GR /EHsc /w /bigobj -openmp -DNDEBUG -DUSE_FBGEMM -DUSE_VULKAN_WRAPPER
--   Build type            : Release
--   Compile definitions   : ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;_CRT_SECURE_NO_DEPRECATE=1;WIN32_LEAN_AND_MEAN;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS
--   CMAKE_PREFIX_PATH     : C:\Users\cbarak\.conda\envs\pytorch_1.7.1\Lib\site-packages;C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1
--   CMAKE_INSTALL_PREFIX  : C:/dev/pytorch/torch
--
--   TORCH_VERSION         : 1.7.0
--   CAFFE2_VERSION        : 1.7.0
--   BUILD_CAFFE2          : ON
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_CAFFE2_MOBILE   : OFF
--   BUILD_STATIC_RUNTIME_BENCHMARK: OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.8.6
--     Python executable   : C:/Users/cbarak/.conda/envs/pytorch_1.7.1/python.exe
--     Pythonlibs version  : 3.8.6
--     Python library      : C:/Users/cbarak/.conda/envs/pytorch_1.7.1/libs/python38.lib
--     Python includes     : C:/Users/cbarak/.conda/envs/pytorch_1.7.1/include
--     Python site-packages: Lib/site-packages
--   BUILD_SHARED_LIBS     : ON
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   BUILD_MOBILE_AUTOGRAD : OFF
--   INTERN_BUILD_MOBILE   :
--   USE_ASAN              : OFF
--   USE_CPP_CODE_COVERAGE : OFF
--   USE_CUDA              : 1
--     CUDA static link    : OFF
--     USE_CUDNN           : ON
--     CUDA version        : 11.1
--     cuDNN version       : 8.0.5
--     CUDA root directory : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1
--     CUDA library        : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/cuda.lib
--     cudart library      : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/cudart_static.lib
--     cublas library      : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/cublas.lib
--     cufft library       : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/cufft.lib
--     curand library      : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/curand.lib
--     cuDNN library       : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/cudnn.lib
--     nvrtc               : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/lib/x64/nvrtc.lib
--     CUDA include path   : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/include
--     NVCC executable     : C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.1/bin/nvcc.exe
--     NVCC flags          : -Xcompiler;/w;-w;-Xfatbin;-compress-all;-DONNX_NAMESPACE=onnx_torch;--use-local-env;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86;-Xcudafe;--diag_suppress=cc_clobber_ignored;-Xcudafe;--diag_suppress=integer_sign_change;-Xcudafe;--diag_suppress=useless_using_declaration;-Xcudafe;--diag_suppress=set_but_not_used;-Xcudafe;--diag_suppress=field_without_dll_interface;-Xcudafe;--diag_suppress=base_class_has_different_dll_interface;-Xcudafe;--diag_suppress=dll_interface_conflict_none_assumed;-Xcudafe;--diag_suppress=dll_interface_conflict_dllexport_assumed;-Xcudafe;--diag_suppress=implicit_return_from_non_void_function;-Xcudafe;--diag_suppress=unsigned_compare_with_zero;-Xcudafe;--diag_suppress=declared_but_not_referenced;-Xcudafe;--diag_suppress=bad_friend_decl;--Werror;cross-execution-space-call;--no-host-device-move-forward;-Xcompiler;-MD$<$<CONFIG:Debug>:d>;--expt-relaxed-constexpr;--expt-extended-lambda;-Xcompiler;/wd4819;-Xcompiler;/wd4503;-Xcompiler;/wd4190;-Xcompiler;/wd4244;-Xcompiler;/wd4251;-Xcompiler;/wd4275;-Xcompiler;/wd4522;-Wno-deprecated-gpu-targets;--expt-extended-lambda;-DCUDA_HAS_FP16=1;-D__CUDA_NO_HALF_OPERATORS__;-D__CUDA_NO_HALF_CONVERSIONS__;-D__CUDA_NO_HALF2_OPERATORS__
--     CUDA host compiler  : C:/Program Files (x86)/Microsoft Visual Studio/2017/Professional/VC/Tools/MSVC/14.16.27023/bin/Hostx64/x64/cl.exe
--     NVCC --device-c     : OFF
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : ON
--   USE_FBGEMM            : ON
--     USE_FAKELOWP          : OFF
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_MKL               : OFF
--   USE_MKLDNN            : ON
--   USE_MKLDNN_CBLAS      : OFF
--   USE_NCCL              : OFF
--   USE_NNPACK            : OFF
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_VULKAN            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : OFF
--   USE_PYTORCH_QNNPACK   : OFF
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI             : OFF
--     USE_GLOO            : ON
--     USE_TENSORPIPE      : OFF
--   Public Dependencies  : Threads::Threads;caffe2::mkldnn
--   Private Dependencies : cpuinfo;fbgemm;fp16;gloo;aten_op_header_gen;foxi_loader;fmt::fmt-header-only
-- Configuring done
-- Generating done
-- Build files have been written to: C:/dev/pytorch/build
cmake --build . --target install --config Release -- -j 12

Context: I am running into the same problems. Has there been a solution yet?


Testing my GPU with looping code example

Question:
Hello everyone
I would like to run python code in loop to test my GPU . Can anyone share a code with me? I don’t have any knowledge in programming and couldn’t fid an example in the net yet .
I’m using Pytorch 1.6 suitable for CUDA 10.2 , Python 3.7.9 and Quadro RTX 4000 . In addition I’m using Spyder 4.2.1 . Host system is based on Windows server 2019
Thank you

Context:
Could you define what you mean by in loop? Is it a training or testing loop?
Assuming you want to benchmark your GPU you can use the this repo

      github.com

ryujaehun/pytorch-gpu-benchmark 15

  Using the famous cnn model in Pytorch, we run benchmarks on various gpu.


If __name__ == ‘__main__’: freeze_support()

Question:
# Imports
import numpy as np
import torch
import torch.utils.data
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
from torch.utils.data.sampler import SubsetRandomSampler
from torchvision import datasets, transforms
import matplotlib.pyplot as plt # for plotting
import time
import pandas as pd

"""Load Data"""

def get_data(path):
    df = pd.read_csv(path, header=None)
    #df = pd.read_csv(path, header=None, usecols=[1,2])
    df = df.astype(np.float32)
    df = df.to_numpy()
    return df.flatten()

def get_data_loader(batch_size):
    data_transforms = transforms.Compose([transforms.ToTensor()])
    path_of_data= "C:\\Users\Stephanie\Documents\ProcessedData\\"
    classes = ["CLOSE", "OPEN"]
    trainset = datasets.DatasetFolder(path_of_data,
                                   loader=get_data, extensions='.csv')

 # Get the list of indices to sample from
    relevant_indices = list(range(0,len(trainset)))

    # Split into train and validation
    np.random.seed(1) # Fixed numpy random seed for reproducible shuffling
    np.random.shuffle(relevant_indices)
    split = int(len(relevant_indices) * 0.8) #split at _%

    # split into training and validation indices
    relevant_train_indices, relevant_val_indices = relevant_indices[:split], relevant_indices[split:]
    train_sampler = SubsetRandomSampler(relevant_train_indices)
    train_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                               num_workers=1, sampler=train_sampler)
    val_sampler = SubsetRandomSampler(relevant_val_indices)
    val_loader = torch.utils.data.DataLoader(trainset, batch_size=batch_size,
                                              num_workers=1, sampler=val_sampler)
    return train_loader, val_loader

def get_model_name(name, batch_size, learning_rate, epoch):
    """ Generate a name for the model consisting of all the hyperparameter values

    Args:
        config: Configuration object containing the hyperparameters
    Returns:
        path: A string with the hyperparameter name and value concatenated
    """
    path = "model_{0}_bs{1}_lr{2}_epoch{3}".format(name,
                                                   batch_size,
                                                   learning_rate,
                                                   epoch)
    return path

def get_accuracy2(model, train=False):

    if train:
        data_loader = train_loader
    else:
        data_loader = val_loader

    correct = 0
    total = 0

    for eeg, labels in data_loader:

        #############################################
        #To Enable GPU Usage
        if use_cuda and torch.cuda.is_available():
          eeg = eeg.cuda()
          labels = labels.cuda()
        #############################################

        transfer_out = encoder(eeg)
        output = model(transfer_out)
        #print(output)

        #select index with maximum prediction score
        pred = output.max(1, keepdim=True)[1]
        correct += pred.eq(labels.view_as(pred)).sum().item()
        total += eeg.shape[0]
    return correct / total

def train_net3(net, split, batch_size=64, learning_rate=0.01, num_epochs=7):

    ########################################################################
    torch.manual_seed(1)

    # Train the network
    # Loop over the data iterator and sample a new batch of training data
    # Get the output from the network, and optimize our loss function.

    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)

    iters, losses, train_acc, val_acc = [], [], [], []

    # training
    n = 0 # the number of iterations
    start_time=time.time()

    for epoch in range(num_epochs):

        for eeg, labels in iter(train_loader):

            #############################################
            #To Enable GPU Usage
            if use_cuda and torch.cuda.is_available():
              eeg = eeg.cuda()
              labels = labels.cuda()
            #############################################
            transfer_out = encoder(eeg)
            #print(transfer_out.shape)
            out = net(transfer_out)
            #print(out)
            #print(labels)
            loss = criterion(out, labels.long())
            #print(loss)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

           ###########################

          # save the current training information
            iters.append(n)
            losses.append(float(loss)/batch_size)             # compute *average* loss
            val_acc.append(get_accuracy2(net, train=False))  # compute validation accuracy
            train_acc.append(get_accuracy2(net, train=True))
            n += 1
            #print("Iteration: ",n,'Progress: % 6.2f ' % ((epoch * len(train_loader) + mini_b) / (num_epochs * len(train_loader))*100),'%', "Time Elapsed: % 6.2f s " % (time.time()-start_time))

        print ("Epoch %d Finished. " % epoch ,"Time per Epoch: % 6.2f s "% ((time.time()-start_time) / (epoch +1)))
        print("Training accuracy:", train_acc[-1], "Validation accuracy:", val_acc[-1])
    # Save the current model (checkpoint) to a file
    model_path = get_model_name(net.name, batch_size, learning_rate, epoch)
    model_path = f"/content/gdrive/My Drive/{model_path}"
    torch.save(net.state_dict(), model_path)

    end_time= time.time()

    train_acc.append(get_accuracy2(net, train=True))
    print("Final Training Accuracy: {}".format(train_acc[-1]))
    print("Final Validation Accuracy: {}".format(val_acc[-1]))
    print ("Total time:  % 6.2f s  Time per Epoch: % 6.2f s " % ( (end_time-start_time), ((end_time-start_time) / num_epochs) ))
    total_acc = train_acc[-1] * split + val_acc[-1] * (1-split)
    print("Accuracy of entire set: {}".format(total_acc))

    # Write the train/test loss/err into CSV file for plotting later
    epochs = np.arange(1, num_epochs + 1)
    np.savetxt("{}_train_acc.csv".format(model_path), train_acc)
    np.savetxt("{}_val_acc.csv".format(model_path), val_acc)

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.name = 'Encoder'
        self.encoder = nn.Sequential(
            nn.Linear(4400, 15) #
        )
        self.decoder = nn.Sequential(
            nn.Linear(15, 4400), #
            nn.Sigmoid() # get to the range (0, 1)
        )

    def forward(self, x):
        x1 = self.encoder(x)
        x2 = self.decoder(x1)
        x1 = F.relu(x1)
        #print(x1.shape)
        return x1  # return encoder

class SmallNet2(nn.Module):
    def __init__(self):
        super(SmallNet2, self).__init__()
        self.name = "Small"
        self.layer1 = nn.Linear(15, 12)
        self.layer2 = nn.Linear(12, 2)
    def forward(self, transfer_out):
        shape = transfer_out.shape
        if shape[1] == 15:
          activation1 = self.layer1(transfer_out.float())
          activation1 = F.relu(activation1)
          activation2 = self.layer2(activation1)
          return activation2

torch.manual_seed(1)

use_cuda = True
## best for all = batch size = 48, lr = 0.005, num epoch = 45
batch_size = 48
lr = 0.005
num_epochs = 45
train_loader, val_loader = get_data_loader(batch_size)
small_net = SmallNet2()
encoder = Encoder()
path = "C:\\Users\Stephanie\PycharmProjects\capstone\Autoencoder_model2"
encoder.load_state_dict(torch.load(path))
encoder.eval()

if use_cuda and torch.cuda.is_available():
  print("CUDA is available")
  small_net = small_net.cuda()
  encoder = encoder.cuda()
else:
    print("CUDA is not available")

if __name__ == '__main__':
    train_net3(small_net, 0.8, batch_size, lr, num_epochs)

If my num_workers is greater than 0, I will get an error without the “if name == ‘main’:”, but when I use this line, the code does not work as intended and training takes 10x longer. Why can I not set num_workers to a number other than 0? I copied this code from my Google Colab file and it worked fine there, but on PyCharm it doesn’t work.
Please help.

Context:
scumzilla:

when I use this line, the code does not work as intended and training takes 10x longer.

In WIndows, if name == ‘_main__’ is needed for multiprocessing support.
However, I believe it should not affect the training time.
If the training takes 10x longer, it may have to do with the type of GPU you are using in your PC vs. in Google Colab.


Why the inference speed differs a lot between windows and linux on pytorch-cpu

Question:
why the inference speed differs a lot between windows and linux on pytorch-cpu?
linux is faster than windows using the same pytorch cpu version without cuda?
why why why!
really important to me!
hope i can get the answer

Context: There is some inherent windows problem with either L2 cache or memory page management, this difference is not specific to pytorch. Also, your builds may be not identical in regards to MKL, avx, openmp threading.


Question:
I was trying to set up torch on my pc. I’m using Windows 10 with python 3.8.6.
I installed torch with this command
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html.
Because I didn’t install CUDA I got False as output at the command torch.cuda.is_available()
But now I don’t know which is the right driver and CUDA-Version for my NVIDIA GeForce 920M.
Can someone please tell me which versions I have to install?

Context:
You’ve installed the CPU pip wheel, so the CUDA runtime isn’t shipped.
That being said, your GPU uses a compute capability of 3.5, which isn’t supported in the pre-built binaries anymore (min. architecture is 3.7), so you would need to build from source using these instructions 131.


Changed to 64bit Python, Still Can’t Install Torch

Question:
I know this is a common issue but when I try to run the command pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html, I get the error:
C:\Users\robsa\AppData\Local\Programs\Python\Python39\Scripts>pip install torch==1.6.0+cpu torchvision==0.7.0+cpu -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.6.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.6.0+cpu

I was initially using 32bit Python but uninstalled that, and installed Python 3.9 64 bit. I still get the same error.
I have been using MSYS2 to compile C++ code. I think this comes with a version of Python as when I run ‘python -v’ in cmd it shows a lot of file paths pointing to a Python installation inside the MSYS2 folder.
Here is what happens when I run pip -V:
C:\Users\robsa\AppData\Local\Programs\Python\Python39\Scripts>pip -V
pip 20.2.4 from c:\users\robsa\appdata\local\programs\python\python39\lib\site-packages\pip (python 3.9)

Running pip -V from a folder other than scripts:
c:\>pip -V
pip 20.2.4 from C:\msys64\mingw32\lib\python3.8\site-packages\pip (python 3.8)

Cheers for any help in advance!

Context:
Did you find anything? I’m facing the same problem.
problems972×510 17.5 KB


Problems with Pytorch file version of reading

Question:
Tried to uninstall and reinstall several different versions from the attempted solution in other threads.
But still encounter this issue of not getting the “supported version for reading”.
Wonder if it is due to the compatibility of windows?
cause it still happens even if I installed the latest version
in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))

RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED
at ..\caffe2\serialize\inline_container.cc:132, please report a bug to PyTorch.
Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2.
Your PyTorch installation may be too old. (init at ..\caffe2\serialize\inline_container.cc:132)
(no backtrace available)

windows 10 1909 (18363.1016)
CPU: AMD 4800HS
GPU: 1650Ti
Pytorch version:1.4.0+cu92
torchvision version: 0.5.0
Python version: 3.7
Compiler: Spyder(anaconda 3)
import syft

Context:
Lee_Shyu:

Pytorch version:1.4.0+cu92

The latest version is 1.6.0 at the moment and soon 1.7 so could you update it and rerun the code?


Question:
After upgrade to rtx3090, I reinstalled my Windows10 and pytorch with

conda install pytorch torchvision cudatoolkit=11 -c pytorch-nightly

When I use nn.Conv/nn.BatchNorm layer in my module and do the forward propagation, it shows

Could not load library cudnn_ops_infer64_8.dll. Error code 126
Please make sure cudnn_ops_infer64_8.dll is in your library path!

in Vscode

Process finished with exit code -1073740791 (0xC0000409)

in Pycharm
I’ve tried reinstalling pytorch and installed cuda toolkit 11 from developer.nvidia.com 10, but the problem remains unsolved.
import torch
import torch.nn as nn

class test(nn.Module):
    def __init__(self):
        super(test, self).__init__()
        self.l = nn.Conv2d(1,1,3)

    def forward(self, input):
        return self.l(input)

net = test().cuda()
print(net(torch.rand(1,1,5,5).cuda()))

Context: Double post from here 556 with follow-up.


Getting C1001 Internal compiler error when building pytorch on windows

Question:
hello everyone.
Is building Pytorch on windows using visual studio 2019 supported?

Context:
OK, This is what I have done to build pytorch on windows. Using Visual Studio 2019 enterprise (16.7.2) :
git clone --branch v1.6.0 https://github.com/pytorch/pytorch.git pytorch-1.6.0
cd pytorch-1.6.0
git submodule update --init --recursive
set CUDAHOSTCXX=
set DISTUTILS_USE_SDK=1
for /f "usebackq tokens=*" %i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=14.27
python setup.py build --cmake-only
cmake-gui

and this is what I did afterward:
pytorch_build1560×732 3.1 MB
I managed to open the visual studio and nearly build everything. but out of 46 targets, 11 failed with these errors.
I dont have any cuda installed and I’m using the cpu version.
Here are the errros I get:
build outout:
1>------ Build started: Project: defs.bzl, Configuration: Release x64 ------
2>------ Build started: Project: ATEN_CPU_FILES_GEN_TARGET, Configuration: Release x64 ------
3>------ Build started: Project: dnnl_cpu, Configuration: Release x64 ------
4>------ Build started: Project: generate-torch-sources, Configuration: Release x64 ------
5>------ Build started: Project: python_copy_files, Configuration: Release x64 ------
6>------ Build started: Project: gen_onnx_proto, Configuration: Release x64 ------
4>Generating ../../torch/csrc/autograd/generated/Functions.cpp, ../../torch/csrc/jit/generated/generated_unboxing_wrappers_0.cpp, ../../torch/csrc/jit/generated/generated_unboxing_wrappers_1.cpp, ../../torch/csrc/jit/generated/generated_unboxing_wrappers_2.cpp, ../../torch/csrc/autograd/generated/VariableType_0.cpp, ../../torch/csrc/autograd/generated/VariableType_1.cpp, ../../torch/csrc/autograd/generated/VariableType_2.cpp, ../../torch/csrc/autograd/generated/VariableType_3.cpp, ../../torch/csrc/autograd/generated/VariableType_4.cpp, ../../torch/csrc/autograd/generated/ProfiledType_0.cpp, ../../torch/csrc/autograd/generated/ProfiledType_1.cpp, ../../torch/csrc/autograd/generated/ProfiledType_2.cpp, ../../torch/csrc/autograd/generated/ProfiledType_3.cpp, ../../torch/csrc/autograd/generated/ProfiledType_4.cpp, ../../torch/csrc/autograd/generated/TraceType_0.cpp, ../../torch/csrc/autograd/generated/TraceType_1.cpp, ../../torch/csrc/autograd/generated/TraceType_2.cpp, ../../torch/csrc/autograd/generated/TraceType_3.cpp, ../../torch/csrc/autograd/generated/TraceType_4.cpp, ../../torch/csrc/autograd/generated/Functions.h, ../../torch/csrc/autograd/generated/variable_factories.h, ../../torch/csrc/autograd/generated/VariableType.h, ../../torch/csrc/autograd/generated/python_functions.cpp, ../../torch/csrc/autograd/generated/python_variable_methods.cpp, ../../torch/csrc/autograd/generated/python_torch_functions.cpp, ../../torch/csrc/autograd/generated/python_nn_functions.cpp, ../../torch/csrc/autograd/generated/python_functions.h
7>------ Build started: Project: __aten_op_header_gen, Configuration: Release x64 ------
8>------ Build started: Project: torch_python_stubs, Configuration: Release x64 ------
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>cpu_concat.cpp
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>cpu_layer_normalization_list.cpp
3>cpu_shuffle_list.cpp
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>gemm_convolution_utils.cpp
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>ref_convolution.cpp
3>ref_deconvolution.cpp
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>ref_eltwise.cpp
3>cl : command line warning D9025: overriding '/W1' with '/w'
3>ref_inner_product.cpp
3>ref_layer_normalization.cpp
3>ref_lrn.cpp
3>ref_pooling.cpp
3>ref_resampling.cpp
3>ref_shuffle.cpp
3>ref_softmax.cpp
3>D:\Codes\pytorch-1.6.0\third_party\ideep\mkl-dnn\src\cpu\gemm_convolution_utils.cpp(401): fatal error C1001: Internal compiler error.
3>(compiler file 'd:\agent\_work\7\s\src\vctools\Compiler\Utc\src\p2\main.c', line 195)
3> To work around this problem, try simplifying or changing the program near the locations listed above.
3>If possible please provide a repro here: https://developercommunity.visualstudio.com
3>Please choose the Technical Support command on the Visual C++
3> Help menu, or open the Technical Support help file for more information
3>  CL!RaiseException()+0x69
3>  CL!RaiseException()+0x69
3>  CL!CloseTypeServerPDB()+0x22e6b
3>  CL!CloseTypeServerPDB()+0xcd30a
3>simple_concat.cpp
3>simple_layer_normalization.cpp
3>simple_layer_normalization_kernels.cpp
3>simple_resampling.cpp
3>simple_sum.cpp
3>gemm_utils_f32.cpp
3>ref_gemm_f32.cpp
3>gemm.cpp
3>gemm_pack.cpp
3>ref_gemm_s8x8s32.cpp
3>simple_gemm_s8s8s32.cpp
3>cpu_matmul_list.cpp
3>cl : command line error D8040: error creating or communicating with child process
3>Done building project "dnnl_cpu.vcxproj" -- FAILED.
9>------ Build started: Project: dnnl, Configuration: Release x64 ------
9>Microsoft (R) Library Manager Version 14.27.29111.0
9>Copyright (C) Microsoft Corporation.  All rights reserved.
9>
9>"/OUT:D:\Codes\pytorch-1.6.0\build_dir\lib\Release\dnnl.lib" /NOLOGO /MACHINE:X64 /machine:x64 /ignore:4049 /ignore:4217 dnnl.dir\Release\version.res
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\batch_normalization.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\binary.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\concat.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\convolution.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\convolution_pd.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\deconvolution.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\dnnl_debug.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\dnnl_debug_autogenerated.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\eltwise.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\engine.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\inner_product.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\layer_normalization.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\lrn.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\matmul.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\memory.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\memory_debug.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\memory_desc_wrapper.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\memory_storage.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\memory_zero_pad.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\pooling.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive_attr.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive_cache.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive_desc.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive_exec_types.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive_hashing.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\primitive_iterator.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\query.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\reorder.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\resampling.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\rnn.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\rw_mutex.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\scratchpad.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\scratchpad_debug.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\shuffle.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\softmax.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\stream.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\stream_attr.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\sum.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\utils.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\common\dnnl_common.dir\Release\verbose.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\bfloat16.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_batch_normalization_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_batch_normalization_utils.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_binary_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_concat.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_convolution_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_deconvolution_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_eltwise_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_engine.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_inner_product_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_layer_normalization_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_lrn_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_pooling_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_reorder.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_resampling_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_rnn_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_shuffle_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_softmax_list.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\cpu_sum.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\gemm_convolution.obj"
9>"D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\gemm_convolution_utils.obj"
9>D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\gemm_convolution_utils.obj : fatal error LNK1136: invalid or corrupt file
9>Done building project "dnnl.vcxproj" -- FAILED.
10>------ Build started: Project: torch_cpu, Configuration: Release x64 ------
11>------ Build started: Project: compat_libs, Configuration: Release x64 ------
11>Error copying file "D:/Codes/pytorch-1.6.0/build_dir/lib/Release/dnnl.lib" to "D:/Codes/pytorch-1.6.0/build_dir/lib/Release/mkldnn.lib".
11>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(234,5): error MSB6006: "cmd.exe" exited with code 1.
11>Done building project "compat_libs.vcxproj" -- FAILED.
10>Generating ../../torch/csrc/autograd/generated/Functions.cpp, ../../torch/csrc/jit/generated/generated_unboxing_wrappers_0.cpp, ../../torch/csrc/jit/generated/generated_unboxing_wrappers_1.cpp, ../../torch/csrc/jit/generated/generated_unboxing_wrappers_2.cpp, ../../torch/csrc/autograd/generated/VariableType_0.cpp, ../../torch/csrc/autograd/generated/VariableType_1.cpp, ../../torch/csrc/autograd/generated/VariableType_2.cpp, ../../torch/csrc/autograd/generated/VariableType_3.cpp, ../../torch/csrc/autograd/generated/VariableType_4.cpp, ../../torch/csrc/autograd/generated/ProfiledType_0.cpp, ../../torch/csrc/autograd/generated/ProfiledType_1.cpp, ../../torch/csrc/autograd/generated/ProfiledType_2.cpp, ../../torch/csrc/autograd/generated/ProfiledType_3.cpp, ../../torch/csrc/autograd/generated/ProfiledType_4.cpp, ../../torch/csrc/autograd/generated/TraceType_0.cpp, ../../torch/csrc/autograd/generated/TraceType_1.cpp, ../../torch/csrc/autograd/generated/TraceType_2.cpp, ../../torch/csrc/autograd/generated/TraceType_3.cpp, ../../torch/csrc/autograd/generated/TraceType_4.cpp, ../../torch/csrc/autograd/generated/Functions.h, ../../torch/csrc/autograd/generated/variable_factories.h, ../../torch/csrc/autograd/generated/VariableType.h, ../../torch/csrc/autograd/generated/python_functions.cpp, ../../torch/csrc/autograd/generated/python_variable_methods.cpp, ../../torch/csrc/autograd/generated/python_torch_functions.cpp, ../../torch/csrc/autograd/generated/python_nn_functions.cpp, ../../torch/csrc/autograd/generated/python_functions.h
4>Skipped writing torch/csrc\autograd\generated\python_functions.h
4>Skipped writing torch/csrc\autograd\generated\python_functions.cpp
4>Skipped writing torch/csrc\autograd\generated\python_variable_methods.cpp
4>Skipped writing torch/csrc\autograd\generated\python_torch_functions.cpp
4>Skipped writing torch/csrc\autograd\generated\python_nn_functions.cpp
4>Skipped writing torch/csrc\autograd\generated\VariableType.h
4>Skipped writing torch/csrc\autograd\generated\VariableType_0.cpp
4>Skipped writing torch/csrc\autograd\generated\ProfiledType_0.cpp
4>Skipped writing torch/csrc\autograd\generated\TraceType_0.cpp
4>Skipped writing torch/csrc\autograd\generated\VariableType_1.cpp
4>Skipped writing torch/csrc\autograd\generated\ProfiledType_1.cpp
4>Skipped writing torch/csrc\autograd\generated\TraceType_1.cpp
4>Skipped writing torch/csrc\autograd\generated\VariableType_2.cpp
4>Skipped writing torch/csrc\autograd\generated\ProfiledType_2.cpp
4>Skipped writing torch/csrc\autograd\generated\TraceType_2.cpp
4>Skipped writing torch/csrc\autograd\generated\VariableType_3.cpp
4>Skipped writing torch/csrc\autograd\generated\ProfiledType_3.cpp
4>Skipped writing torch/csrc\autograd\generated\TraceType_3.cpp
4>Skipped writing torch/csrc\autograd\generated\VariableType_4.cpp
4>Skipped writing torch/csrc\autograd\generated\ProfiledType_4.cpp
4>Skipped writing torch/csrc\autograd\generated\TraceType_4.cpp
4>Skipped writing torch/csrc\autograd\generated\VariableTypeEverything.cpp
4>Skipped writing torch/csrc\autograd\generated\ProfiledTypeEverything.cpp
4>Skipped writing torch/csrc\autograd\generated\TraceTypeEverything.cpp
4>Skipped writing torch/csrc\autograd\generated\RegistrationDeclarations.h
4>Skipped writing torch/csrc\autograd\generated\Functions.h
4>Skipped writing torch/csrc\autograd\generated\Functions.cpp
4>Skipped writing torch/csrc\autograd\generated\variable_factories.h
4>Skipped writing torch/csrc\jit\generated\generated_unboxing_wrappers_0.cpp
4>Skipped writing torch/csrc\jit\generated\generated_unboxing_wrappers_1.cpp
4>Skipped writing torch/csrc\jit\generated\generated_unboxing_wrappers_2.cpp
10>Skipped writing torch/csrc\autograd\generated\python_functions.h
10>Skipped writing torch/csrc\autograd\generated\python_functions.cpp
10>Skipped writing torch/csrc\autograd\generated\python_variable_methods.cpp
10>Skipped writing torch/csrc\autograd\generated\python_torch_functions.cpp
10>Skipped writing torch/csrc\autograd\generated\python_nn_functions.cpp
10>Skipped writing torch/csrc\autograd\generated\VariableType.h
10>Skipped writing torch/csrc\autograd\generated\VariableType_0.cpp
10>Skipped writing torch/csrc\autograd\generated\ProfiledType_0.cpp
10>Skipped writing torch/csrc\autograd\generated\TraceType_0.cpp
10>Skipped writing torch/csrc\autograd\generated\VariableType_1.cpp
10>Skipped writing torch/csrc\autograd\generated\ProfiledType_1.cpp
10>Skipped writing torch/csrc\autograd\generated\TraceType_1.cpp
10>Skipped writing torch/csrc\autograd\generated\VariableType_2.cpp
10>Skipped writing torch/csrc\autograd\generated\ProfiledType_2.cpp
10>Skipped writing torch/csrc\autograd\generated\TraceType_2.cpp
10>Skipped writing torch/csrc\autograd\generated\VariableType_3.cpp
10>Skipped writing torch/csrc\autograd\generated\ProfiledType_3.cpp
10>Skipped writing torch/csrc\autograd\generated\TraceType_3.cpp
10>Skipped writing torch/csrc\autograd\generated\VariableType_4.cpp
10>Skipped writing torch/csrc\autograd\generated\ProfiledType_4.cpp
10>Skipped writing torch/csrc\autograd\generated\TraceType_4.cpp
10>Skipped writing torch/csrc\autograd\generated\VariableTypeEverything.cpp
10>Skipped writing torch/csrc\autograd\generated\ProfiledTypeEverything.cpp
10>Skipped writing torch/csrc\autograd\generated\TraceTypeEverything.cpp
10>Skipped writing torch/csrc\autograd\generated\RegistrationDeclarations.h
10>Skipped writing torch/csrc\autograd\generated\Functions.h
10>Skipped writing torch/csrc\autograd\generated\Functions.cpp
10>Skipped writing torch/csrc\autograd\generated\variable_factories.h
10>Skipped writing torch/csrc\jit\generated\generated_unboxing_wrappers_0.cpp
10>Skipped writing torch/csrc\jit\generated\generated_unboxing_wrappers_1.cpp
10>Skipped writing torch/csrc\jit\generated\generated_unboxing_wrappers_2.cpp
10>LINK : fatal error LNK1181: cannot open input file '..\lib\Release\dnnl.lib'
10>Done building project "torch_cpu.vcxproj" -- FAILED.
12>------ Build started: Project: torch, Configuration: Release x64 ------
12>LINK : fatal error LNK1181: cannot open input file '..\lib\Release\torch_cpu.lib'
12>Done building project "torch.vcxproj" -- FAILED.
13>------ Build started: Project: caffe2_pybind11_state, Configuration: Release x64 ------
14>------ Build started: Project: shm, Configuration: Release x64 ------
15>------ Build started: Project: caffe2_observers, Configuration: Release x64 ------
16>------ Build started: Project: caffe2_detectron_ops, Configuration: Release x64 ------
15>LINK : fatal error LNK1181: cannot open input file '..\..\lib\Release\torch.lib'
15>Done building project "caffe2_observers.vcxproj" -- FAILED.
14>LINK : fatal error LNK1181: cannot open input file '..\..\..\..\lib\Release\torch.lib'
14>Done building project "shm.vcxproj" -- FAILED.
17>------ Build started: Project: torch_python, Configuration: Release x64 ------
18>------ Build started: Project: windows_python_copy_lib, Configuration: Release x64 ------
16>LINK : fatal error LNK1181: cannot open input file '..\..\lib\Release\torch.lib'
16>Done building project "caffe2_detectron_ops.vcxproj" -- FAILED.
18>Error copying file "D:/Codes/pytorch-1.6.0/build_dir/caffe2/python/Release/caffe2_pybind11_state.cp37-win_amd64.pyd" to "D:/Codes/pytorch-1.6.0/build_dir/caffe2/python".
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: The command "setlocal
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: "C:\Program Files\CMake\bin\cmake.exe" -E copy D:/Codes/pytorch-1.6.0/build_dir/caffe2/python/Release/caffe2_pybind11_state.cp37-win_amd64.pyd D:/Codes/pytorch-1.6.0/build_dir/caffe2/python
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: if %errorlevel% neq 0 goto :cmEnd
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: :cmEnd
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: :cmErrorLevel
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: exit /b %1
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: :cmDone
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: if %errorlevel% neq 0 goto :VCEnd
18>C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets(149,5): error MSB3073: :VCEnd" exited with code 1.
18>Done building project "windows_python_copy_lib.vcxproj" -- FAILED.
17>LINK : fatal error LNK1181: cannot open input file '..\..\lib\Release\shm.lib'
17>Done building project "torch_python.vcxproj" -- FAILED.
========== Build: 7 succeeded, 11 failed, 28 up-to-date, 0 skipped ==========

Linker errrors:
Severity	Code	Description	Project	File	Line	Suppression State
Error	C1001	Internal compiler error.	dnnl_cpu	D:\Codes\pytorch-1.6.0\third_party\ideep\mkl-dnn\src\cpu\gemm_convolution_utils.cpp	401
Error	D8040	error creating or communicating with child process	dnnl_cpu	D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\cl	1
Error	LNK1136	invalid or corrupt file	dnnl	D:\Codes\pytorch-1.6.0\build_dir\third_party\ideep\mkl-dnn\src\cpu\dnnl_cpu.dir\Release\gemm_convolution_utils.obj	1
Error	MSB6006	"cmd.exe" exited with code 1.	compat_libs	C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets	234
Error	LNK1181	cannot open input file '..\lib\Release\dnnl.lib'	torch_cpu	D:\Codes\pytorch-1.6.0\build_dir\caffe2\LINK	1
Error	LNK1181	cannot open input file '..\lib\Release\torch_cpu.lib'	torch	D:\Codes\pytorch-1.6.0\build_dir\caffe2\LINK	1
Error	LNK1181	cannot open input file '..\..\lib\Release\torch.lib'	caffe2_observers	D:\Codes\pytorch-1.6.0\build_dir\modules\observers\LINK	1
Error	LNK1181	cannot open input file '..\..\..\..\lib\Release\torch.lib'	shm	D:\Codes\pytorch-1.6.0\build_dir\caffe2\torch\lib\libshm_windows\LINK	1
Error	LNK1181	cannot open input file '..\..\lib\Release\torch.lib'	caffe2_detectron_ops	D:\Codes\pytorch-1.6.0\build_dir\modules\detectron\LINK	1
Error	MSB3073	The command "setlocal
"C:\Program Files\CMake\bin\cmake.exe" -E copy D:/Codes/pytorch-1.6.0/build_dir/caffe2/python/Release/caffe2_pybind11_state.cp37-win_amd64.pyd D:/Codes/pytorch-1.6.0/build_dir/caffe2/python
if %errorlevel% neq 0 goto :cmEnd
:cmEnd
endlocal & call :cmErrorLevel %errorlevel% & goto :cmDone
:cmErrorLevel
exit /b %1
:cmDone
if %errorlevel% neq 0 goto :VCEnd
:VCEnd" exited with code 1.	windows_python_copy_lib	C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\MSBuild\Microsoft\VC\v160\Microsoft.CppCommon.targets	149
Error	LNK1181	cannot open input file '..\..\lib\Release\shm.lib'	torch_python	D:\Codes\pytorch-1.6.0\build_dir\caffe2\torch\LINK	1

What am I missing here?
please note that this is the default, and I have not yet changed to MT yet. its all MD by default.
any help is greatly apprecaited.
@peterjc123


Problems after installing pytorch on windows

Question:
I have installed pytorch in a conda environment using
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch

However after that I try doing import torch
and I get the following error
Microsoft Visual C++ Redistributable is not installed, this may lead to the DLL load failure.
It can be downloaded at https://aka.ms/vs/16/release/vc_redist.x64.exe
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
File "C:\Users\aliag\anaconda3\envs\pytorchPractice\lib\site-packages\torch\__init__.py", line 127, in <module>
raise err
OSError: [WinError 126] 指定されたモジュールが見つかりません。
Error loading "C:\Users\aliag\anaconda3\envs\pytorchPractice\lib\site-packages\torch\lib\asmjit.dll"
or one of its dependencies.
>>>

I would really appreciate some help to correct this problem

Context: I found the solution. I had not adequately installed CUDA. When I did it again, the problem dissapears


Install problem: pytorch 1.2.0 with Cuda 10.2

Question:
Hi,
I’m trying to run code from the following github repository: https://github.com/AnticipatedLearningMachine/Anticipated-Learning-Machine 1
In the requirements section at the bottom, it says that pytorch 1.2.0 and CUDA 10.2 are required. I have tried installing pytorch 1.2.0 (on Windows Anaconda) via pip, but I get a version that comes with Cuda 9.2. Can someone please help me install pytorch 1.2.0 with CUDA 10.2?
Thanks,
Gavin.

Context:
I don’t think that the PyTorch 1.2.0 binaries and wheels were built with CUDA10.2, but with CUDA9.2 and CUDA10.0 as seen here 17.
If you need this config for a particular reason, you would most likely have to build PyTorch from source.


ModuleNotFoundError: No module named ‘torch’ in IDLE

Question:
So I installed pytorch through anaconda successfully and when I try importing torch in IDLE I get this error
I did what I found on stack exchange “print(sys.executable)”
Conda’s result: C:\Users\readz\anaconda3\python.exe
IDLE’s result: C:\Users\readz\AppData\Local\Programs\Python\Python38-32\pythonw.exe
Problem is there they didn’t explain what to do with this info or how to solve the problem…

Context:
Did you install IDLE in the same conda environment and executed it there?
If not, could you do it, as currently IDLE seems to use another Python environment.


Question:
Hello, everyone.
I want to run Deep Learning model for multivariate time series.
For example, below is the daily delivery amount of post office
delivery date, post office id, delivery amount, weekday, …
which is daily data, multivariate
I want to predict future delivery amount using data above.
Can I run this as deep learning model using LSTM??
That is, can LSTM run mutlivariate time series data for prediction??
If it is not possible, which deep learning model should I use??
Thank you in advance and have a nice day!

Context:
morphism:

Can I run this as deep learning model using LSTM??

Yes you can use LSTM for time series data prediction. You can find alot of resources for that purpose.
You can check this github repo for research papers and link for data resources.
https://github.com/Alro10/deep-learning-time-series 449
If you want to checkout for implementation you can also find that in below link.
https://github.com/zhangxu0307/time_series_forecasting_pytorch 531
https://github.com/KurochkinAlexey/ConvRNN 147
https://stackoverflow.com/questions/56858924/multivariate-input-lstm-in-pytorch 463


Question:
Hi, I am training my dataset using this code below, in linux it runs well and fast but when I try to run on windows I can only run if num-workers = 0 but that makes the process much slower. Does anyone know how I can run this code on windows with num-workers> 0?
import os
import sys
import logging
import argparse
import itertools
import torch

from torch.utils.data import DataLoader, ConcatDataset
from torch.optim.lr_scheduler import CosineAnnealingLR, MultiStepLR

from vision.utils.misc import str2bool, Timer, freeze_net_layers, store_labels
from vision.ssd.ssd import MatchPrior
from vision.ssd.vgg_ssd import create_vgg_ssd
from vision.ssd.mobilenetv1_ssd import create_mobilenetv1_ssd
from vision.ssd.mobilenetv1_ssd_lite import create_mobilenetv1_ssd_lite
from vision.ssd.mobilenet_v2_ssd_lite import create_mobilenetv2_ssd_lite
from vision.ssd.squeezenet_ssd_lite import create_squeezenet_ssd_lite
from vision.datasets.voc_dataset import VOCDataset
from vision.datasets.open_images import OpenImagesDataset
from vision.nn.multibox_loss import MultiboxLoss
from vision.ssd.config import vgg_ssd_config
from vision.ssd.config import mobilenetv1_ssd_config
from vision.ssd.config import squeezenet_ssd_config
from vision.ssd.data_preprocessing import TrainAugmentation, TestTransform

parser = argparse.ArgumentParser(
    description='Single Shot MultiBox Detector Training With PyTorch')

# Params for datasets
parser.add_argument("--dataset-type", default="open_images", type=str,
                    help='Specify dataset type. Currently supports voc and open_images.')
parser.add_argument('--datasets', '--data', nargs='+', default=["data"], help='Dataset directory path')
parser.add_argument('--balance-data', action='store_true',
                    help="Balance training data by down-sampling more frequent labels.")

# Params for network
parser.add_argument('--net', default="mb1-ssd",
                    help="The network architecture, it can be mb1-ssd, mb1-lite-ssd, mb2-ssd-lite or vgg16-ssd.")
parser.add_argument('--freeze-base-net', action='store_true',
                    help="Freeze base net layers.")
parser.add_argument('--freeze-net', action='store_true',
                    help="Freeze all the layers except the prediction head.")
parser.add_argument('--mb2-width-mult', default=1.0, type=float,
                    help='Width Multiplifier for MobilenetV2')

# Params for loading pretrained basenet or checkpoints.
parser.add_argument('--base-net', help='Pretrained base model')
parser.add_argument('--pretrained-ssd', default='models/mobilenet-v1-ssd-mp-0_675.pth', type=str, help='Pre-trained base model')
parser.add_argument('--resume', default=None, type=str,
                    help='Checkpoint state_dict file to resume training from')

# Params for SGD
parser.add_argument('--lr', '--learning-rate', default=0.01, type=float,
                    help='initial learning rate')
parser.add_argument('--momentum', default=0.9, type=float,
                    help='Momentum value for optim')
parser.add_argument('--weight-decay', default=5e-4, type=float,
                    help='Weight decay for SGD')
parser.add_argument('--gamma', default=0.1, type=float,
                    help='Gamma update for SGD')
parser.add_argument('--base-net-lr', default=0.001, type=float,
                    help='initial learning rate for base net, or None to use --lr')
parser.add_argument('--extra-layers-lr', default=None, type=float,
                    help='initial learning rate for the layers not in base net and prediction heads.')

# Scheduler
parser.add_argument('--scheduler', default="cosine", type=str,
                    help="Scheduler for SGD. It can one of multi-step and cosine")

# Params for Multi-step Scheduler
parser.add_argument('--milestones', default="80,100", type=str,
                    help="milestones for MultiStepLR")

# Params for Cosine Annealing
parser.add_argument('--t-max', default=100, type=float,
                    help='T_max value for Cosine Annealing Scheduler.')

# Train params
parser.add_argument('--batch-size', default=4, type=int,
                    help='Batch size for training')
parser.add_argument('--num-epochs', default=30, type=int,
                    help='the number epochs')
parser.add_argument('--num-workers', default=2, type=int,
                    help='Number of workers used in dataloading')
parser.add_argument('--validation-epochs', default=1, type=int,
                    help='the number epochs between running validation')
parser.add_argument('--debug-steps', default=10, type=int,
                    help='Set the debug log output frequency.')
parser.add_argument('--use-cuda', default=True, type=str2bool,
                    help='Use CUDA to train model')
parser.add_argument('--checkpoint-folder', '--model-dir', default='models/',
                    help='Directory for saving checkpoint models')

logging.basicConfig(stream=sys.stdout, level=logging.INFO,
                    format='%(asctime)s - %(message)s', datefmt="%Y-%m-%d %H:%M:%S")

args = parser.parse_args()
DEVICE = torch.device("cuda:0" if torch.cuda.is_available() and args.use_cuda else "cpu")

if args.use_cuda and torch.cuda.is_available():
    torch.backends.cudnn.benchmark = True
    logging.info("Using CUDA...")

def train(loader, net, criterion, optimizer, device, debug_steps=100, epoch=-1):
    net.train(True)
    running_loss = 0.0
    running_regression_loss = 0.0
    running_classification_loss = 0.0
    for i, data in enumerate(loader):
        images, boxes, labels = data
        images = images.to(device)
        boxes = boxes.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()
        confidence, locations = net(images)
        regression_loss, classification_loss = criterion(confidence, locations, labels, boxes)  # TODO CHANGE BOXES
        loss = regression_loss + classification_loss
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        running_regression_loss += regression_loss.item()
        running_classification_loss += classification_loss.item()
        if i and i % debug_steps == 0:
            avg_loss = running_loss / debug_steps
            avg_reg_loss = running_regression_loss / debug_steps
            avg_clf_loss = running_classification_loss / debug_steps
            logging.info(
                f"Epoch: {epoch}, Step: {i}/{len(loader)}, " +
                f"Avg Loss: {avg_loss:.4f}, " +
                f"Avg Regression Loss {avg_reg_loss:.4f}, " +
                f"Avg Classification Loss: {avg_clf_loss:.4f}"
            )
            running_loss = 0.0
            running_regression_loss = 0.0
            running_classification_loss = 0.0

def test(loader, net, criterion, device):
    net.eval()
    running_loss = 0.0
    running_regression_loss = 0.0
    running_classification_loss = 0.0
    num = 0
    for _, data in enumerate(loader):
        images, boxes, labels = data
        images = images.to(device)
        boxes = boxes.to(device)
        labels = labels.to(device)
        num += 1

        with torch.no_grad():
            confidence, locations = net(images)
            regression_loss, classification_loss = criterion(confidence, locations, labels, boxes)
            loss = regression_loss + classification_loss

        running_loss += loss.item()
        running_regression_loss += regression_loss.item()
        running_classification_loss += classification_loss.item()
    return running_loss / num, running_regression_loss / num, running_classification_loss / num

if __name__ == '__main__':
    timer = Timer()

    logging.info(args)

    # make sure that the checkpoint output dir exists
    if args.checkpoint_folder:
        args.checkpoint_folder = os.path.expanduser(args.checkpoint_folder)

        if not os.path.exists(args.checkpoint_folder):
            os.mkdir(args.checkpoint_folder)

    # select the network architecture and config
    if args.net == 'vgg16-ssd':
        create_net = create_vgg_ssd
        config = vgg_ssd_config
    elif args.net == 'mb1-ssd':
        create_net = create_mobilenetv1_ssd
        config = mobilenetv1_ssd_config
    elif args.net == 'mb1-ssd-lite':
        create_net = create_mobilenetv1_ssd_lite
        config = mobilenetv1_ssd_config
    elif args.net == 'sq-ssd-lite':
        create_net = create_squeezenet_ssd_lite
        config = squeezenet_ssd_config
    elif args.net == 'mb2-ssd-lite':
        create_net = lambda num: create_mobilenetv2_ssd_lite(num, width_mult=args.mb2_width_mult)
        config = mobilenetv1_ssd_config
    else:
        logging.fatal("The net type is wrong.")
        parser.print_help(sys.stderr)
        sys.exit(1)

    # create data transforms for train/test/val
    train_transform = TrainAugmentation(config.image_size, config.image_mean, config.image_std)
    target_transform = MatchPrior(config.priors, config.center_variance,
                                  config.size_variance, 0.5)

    test_transform = TestTransform(config.image_size, config.image_mean, config.image_std)

    # load datasets (could be multiple)
    logging.info("Prepare training datasets.")
    datasets = []
    for dataset_path in args.datasets:
        if args.dataset_type == 'voc':
            dataset = VOCDataset(dataset_path, transform=train_transform,
                                 target_transform=target_transform)
            label_file = os.path.join(args.checkpoint_folder, "labels.txt")
            store_labels(label_file, dataset.class_names)
            num_classes = len(dataset.class_names)
        elif args.dataset_type == 'open_images':
            dataset = OpenImagesDataset(dataset_path,
                 transform=train_transform, target_transform=target_transform,
                 dataset_type="train", balance_data=args.balance_data)
            label_file = os.path.join(args.checkpoint_folder, "labels.txt")
            store_labels(label_file, dataset.class_names)
            logging.info(dataset)
            num_classes = len(dataset.class_names)

        else:
            raise ValueError(f"Dataset type {args.dataset_type} is not supported.")
        datasets.append(dataset)

    # create training dataset
    logging.info(f"Stored labels into file {label_file}.")
    train_dataset = ConcatDataset(datasets)
    logging.info("Train dataset size: {}".format(len(train_dataset)))
    train_loader = DataLoader(train_dataset, args.batch_size,
                              num_workers=args.num_workers,
                              shuffle=True)

    # create validation dataset
    logging.info("Prepare Validation datasets.")
    if args.dataset_type == "voc":
        val_dataset = VOCDataset(dataset_path, transform=test_transform,
                                 target_transform=target_transform, is_test=True)
    elif args.dataset_type == 'open_images':
        val_dataset = OpenImagesDataset(dataset_path,
                                        transform=test_transform, target_transform=target_transform,
                                        dataset_type="test")
        logging.info(val_dataset)
    logging.info("Validation dataset size: {}".format(len(val_dataset)))

    val_loader = DataLoader(val_dataset, args.batch_size,
                            num_workers=args.num_workers,
                            shuffle=False)

    # create the network
    logging.info("Build network.")
    net = create_net(num_classes)
    min_loss = -10000.0
    last_epoch = -1

    # freeze certain layers (if requested)
    base_net_lr = args.base_net_lr if args.base_net_lr is not None else args.lr
    extra_layers_lr = args.extra_layers_lr if args.extra_layers_lr is not None else args.lr

    if args.freeze_base_net:
        logging.info("Freeze base net.")
        freeze_net_layers(net.base_net)
        params = itertools.chain(net.source_layer_add_ons.parameters(), net.extras.parameters(),
                                 net.regression_headers.parameters(), net.classification_headers.parameters())
        params = [
            {'params': itertools.chain(
                net.source_layer_add_ons.parameters(),
                net.extras.parameters()
            ), 'lr': extra_layers_lr},
            {'params': itertools.chain(
                net.regression_headers.parameters(),
                net.classification_headers.parameters()
            )}
        ]
    elif args.freeze_net:
        freeze_net_layers(net.base_net)
        freeze_net_layers(net.source_layer_add_ons)
        freeze_net_layers(net.extras)
        params = itertools.chain(net.regression_headers.parameters(), net.classification_headers.parameters())
        logging.info("Freeze all the layers except prediction heads.")
    else:
        params = [
            {'params': net.base_net.parameters(), 'lr': base_net_lr},
            {'params': itertools.chain(
                net.source_layer_add_ons.parameters(),
                net.extras.parameters()
            ), 'lr': extra_layers_lr},
            {'params': itertools.chain(
                net.regression_headers.parameters(),
                net.classification_headers.parameters()
            )}
        ]

    # load a previous model checkpoint (if requested)
    timer.start("Load Model")
    if args.resume:
        logging.info(f"Resume from the model {args.resume}")
        net.load(args.resume)
    elif args.base_net:
        logging.info(f"Init from base net {args.base_net}")
        net.init_from_base_net(args.base_net)
    elif args.pretrained_ssd:
        logging.info(f"Init from pretrained ssd {args.pretrained_ssd}")
        net.init_from_pretrained_ssd(args.pretrained_ssd)
    logging.info(f'Took {timer.end("Load Model"):.2f} seconds to load the model.')

    # move the model to GPU
    net.to(DEVICE)

    # define loss function and optimizer
    criterion = MultiboxLoss(config.priors, iou_threshold=0.5, neg_pos_ratio=3,
                             center_variance=0.1, size_variance=0.2, device=DEVICE)
    optimizer = torch.optim.SGD(params, lr=args.lr, momentum=args.momentum,
                                weight_decay=args.weight_decay)
    logging.info(f"Learning rate: {args.lr}, Base net learning rate: {base_net_lr}, "
                 + f"Extra Layers learning rate: {extra_layers_lr}.")

    # set learning rate policy
    if args.scheduler == 'multi-step':
        logging.info("Uses MultiStepLR scheduler.")
        milestones = [int(v.strip()) for v in args.milestones.split(",")]
        scheduler = MultiStepLR(optimizer, milestones=milestones,
                                                     gamma=0.1, last_epoch=last_epoch)
    elif args.scheduler == 'cosine':
        logging.info("Uses CosineAnnealingLR scheduler.")
        scheduler = CosineAnnealingLR(optimizer, args.t_max, last_epoch=last_epoch)
    else:
        logging.fatal(f"Unsupported Scheduler: {args.scheduler}.")
        parser.print_help(sys.stderr)
        sys.exit(1)

    # train for the desired number of epochs
    logging.info(f"Start training from epoch {last_epoch + 1}.")

    for epoch in range(last_epoch + 1, args.num_epochs):
        scheduler.step()
        train(train_loader, net, criterion, optimizer,
              device=DEVICE, debug_steps=args.debug_steps, epoch=epoch)

        if epoch % args.validation_epochs == 0 or epoch == args.num_epochs - 1:
            val_loss, val_regression_loss, val_classification_loss = test(val_loader, net, criterion, DEVICE)
            logging.info(
                f"Epoch: {epoch}, " +
                f"Validation Loss: {val_loss:.4f}, " +
                f"Validation Regression Loss {val_regression_loss:.4f}, " +
                f"Validation Classification Loss: {val_classification_loss:.4f}"
            )
            model_path = os.path.join(args.checkpoint_folder, f"{args.net}-Epoch-{epoch}-Loss-{val_loss}.pth")
            torch.save(net.state_dict(),model_path)
            logging.info(f"Saved model {model_path}")

    logging.info("Task done, exiting program.")

Context:
Based on the FAQ 39 it seems you might need to move the complete code to the if-clause protection.
Could you move the torch.cuda.is_available() calls into the if guard (i.e. the DEVICE assignment and setting cudnn.benchmark)?
If that doesn’t help, could you post the error message?


Help with OSError caffe2_detectron_ops_gpu

Question:
Hello. I keep getting the error

OSError: [WinError 126] The specified module could not be found. Error loading “C:\Users\Username\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\lib\caffe2_detectron_ops_gpu.dll” or one of its dependencies.

I tried updating form cuda 10.1 to 10.2 but it didnt fix the issue. i also have cudnn installed . I uninstalled and reistalled torch multiple times with the commands found on the getting started page. Can someone help me resolve this i’ve being stuck for 2 days now

Context: Problem solved cudnn64_7.dll was missing from system32 folder


Is building from source for windows up to date? (subprocess.CalledProcessError: Command ‘[‘cmake’, ..] returned non-zero exit status 1 error)

Question:
Hi everyone.
I’m in the process of converting our models from Python to C++ for production. For that it seems we need to build Pytorch from source as the prebuilt libtorch doesnt come with torch.h headers and alike !
I’m on Windows 10 1803 and  have Visual Studio 2019 installed (v16.6.3). I dont have any GPUs so I didnt install CUDA! or NVTX.
I have anaconda and python 3.7 installed and all the deps are installed using the following command:
conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi

I cloned Pytorch as stated in the readme :
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
# if you are updating an existing checkout
git submodule sync
git submodule update --init --recursive

But upon executing these commands :
:: [Optional] If you want to build with VS 2019 generator, please change the value in the next line to `Visual Studio 16 2019`.
:: Note: This value is useless if Ninja is detected. However, you can force that by using `set USE_NINJA=OFF`.
set CMAKE_GENERATOR=Visual Studio 16 2019

:: Read the content in the previous section carefully before you proceed.
:: [Optional] If you want to override the underlying toolset used by Ninja and Visual Studio with CUDA, please run the following script block.
:: "Visual Studio 2017 Developer Command Prompt" will be run automatically.
:: Make sure you have CMake >= 3.12 before you do this when you use the Visual Studio generator.
set CMAKE_GENERATOR_TOOLSET_VERSION=14.11
set DISTUTILS_USE_SDK=1
for /f "usebackq tokens=*" %i in (`"%ProgramFiles(x86)%\Microsoft Visual Studio\Installer\vswhere.exe" -version [15^,16^) -products * -latest -property installationPath`) do call "%i\VC\Auxiliary\Build\vcvarsall.bat" x64 -vcvars_ver=%CMAKE_GENERATOR_TOOLSET_VERSION%

:: [Optional] If you want to override the cuda host compiler
set CUDAHOSTCXX=C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\VC\Tools\MSVC\14.11.25503\bin\HostX64\x64\cl.exe

python setup.py install

I face these errors:
D:\Codes\pytorch>python setup.py install
Building wheel torch-1.7.0a0+8376284
-- Building version 1.7.0a0+8376284
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_GENERATOR=Visual Studio 16 2019 -DCMAKE_GENERATOR_TOOLSET_VERSION=14.11 -DCMAKE_INSTALL_PREFIX=D:\Codes\pytorch\torch -DCMAKE_PREFIX_PATH=C:\Users\User\Anaconda3\Lib\site-packages -DNUMPY_INCLUDE_DIR=C:\Users\User\Anaconda3\Lib\site-packages\numpy\core\include -DPYTHON_EXECUTABLE=C:\Users\User\Anaconda3\python.exe -DPYTHON_INCLUDE_DIR=C:\Users\User\Anaconda3\include -DPYTHON_LIBRARY=C:\Users\User\Anaconda3/libs/python37.lib -DTORCH_BUILD_VERSION=1.7.0a0+8376284 -DUSE_NUMPY=True D:\Codes\pytorch
CMake Error: Error: generator : Ninja
Does not match the generator used previously: Visual Studio 16 2019
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.
Traceback (most recent call last):
  File "setup.py", line 737, in <module>
    build_deps()
  File "setup.py", line 321, in build_deps
    cmake=cmake)
  File "D:\Codes\pytorch\tools\build_pytorch_libs.py", line 59, in build_caffe2
    rerun_cmake)
  File "D:\Codes\pytorch\tools\setup_helpers\cmake.py", line 329, in generate
    self.run(args, env=my_env)
  File "D:\Codes\pytorch\tools\setup_helpers\cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "C:\Users\User\Anaconda3\Lib\subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_GENERATOR=Visual Studio 16 2019', '-DCMAKE_GENERATOR_TOOLSET_VERSION=14.11', '-DCMAKE_INSTALL_PREFIX=D:\\Codes\\pytorch\\torch', '-DCMAKE_PREFIX_PATH=C:\\Users\\User\\Anaconda3\\Lib\\site-packages', '-DNUMPY_INCLUDE_DIR=C:\\Users\\User\\Anaconda3\\Lib\\site-packages\\numpy\\core\\include', '-DPYTHON_EXECUTABLE=C:\\Users\\User\\Anaconda3\\python.exe', '-DPYTHON_INCLUDE_DIR=C:\\Users\\User\\Anaconda3\\include', '-DPYTHON_LIBRARY=C:\\Users\\User\\Anaconda3/libs/python37.lib', '-DTORCH_BUILD_VERSION=1.7.0a0+8376284', '-DUSE_NUMPY=True', 'D:\\Codes\\pytorch']' returned non-zero exit status 1.

What am I doing wrong?
Any help is greatly appreciated

Context:
OK, Here are my findings.
First of all include <torch/torch.h> doesnt need building from source. its just in the a different path that is,
in include directories, one need to add both :
$(SolutionDir)Dependencies\libtorch-debug-latest\libtorch\include; $(SolutionDir)Dependencies\libtorch-debug-latest\libtorch\include\torch\csrc\api\include;
and voila you are good to go!


Question:
Hi. I’m new on the deep learning world, so I decided to build a model for pneumonia detection for my country. I’ve selected a well known dataset NIH Chest X-ray 1. It have 14 different classes but I interested only in two classes: Consolidation and Infiltration. I’ll use Data augmentation and transfer learning for the training. Could you please give me an example how train only these 2 clases? I’m thinking about read de csv label database delete images who not have the finding label.
Thank you in advance

Context:
Cesar_Sotelo:

I’m thinking about read de csv label database delete images who not have the finding label.

This sounds like a valid approach.
If you are not interested in training the other classes at the moment, I would either remove them or create a separate smaller dataset, which only contains the desired class samples.


Building pytorch binary classification model

Question:
Hi, I’m building pytorh binary classification model(eg : cat vs dog)
My model’s output is
[[0.4820, 0.5180]] and my lable is [1,0] for example.
my loss is criterion = nn.CrossEntropyLoss()
loss = criterion(outputs, true_value)
#loss = criterion([[0.4820, 0.5180]] , [1,0])

I’m expecting that, the lable is [1,0] than… output shold be [[0.99, 0.01]] like that…
BUT, there is many error or no loss is downgoing…
The datashape of lable and output is not right? please help!!
could you give me the correct shape of lable and outputs?
my shape of outputs is 1x2 and my label’s shape is 2 and batchsize is 1 in my code.
(I want to find pytorch binary classification guide at homepage but I failed T.T)
log:
I got this error.

    loss = criterion(outputs, true_value)
  File "/home/hsm/anaconda3/envs/reproducibleresearch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/hsm/anaconda3/envs/reproducibleresearch/lib/python3.6/site-packages/torch/nn/modules/loss.py", line 1048, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/hsm/anaconda3/envs/reproducibleresearch/lib/python3.6/site-packages/torch/nn/functional.py", line 2690, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/hsm/anaconda3/envs/reproducibleresearch/lib/python3.6/site-packages/torch/nn/functional.py", line 2382, in nll_loss
    "Expected input batch_size ({}) to match target batch_size ({}).".format(input.size(0), target.size(0))
ValueError: Expected input batch_size (1) to match target batch_size (2).

Context:
Adding more info to @larcane 's explanation,

true_value should be a LongTensor which means the type of elements inside the tensor is long.
Your true_value should have only one value. In this case, [0] or [1].
Please make it clear whether your output should be updated into [0, 1] or [1, 0]


Question:
Hello. I am trying to use augmented and not augmented dataset in each epoch(for example: augmented in one epoch not augmented in different epoch) but i couldn’t figure out how to do it. My approach was loading DataLoader in each epoch again and again but I think it’s wrong. Because when i print indexes in getitem in my Dataset, there are a lot of duplicate indexes.
Here is my code for training: CodePile | Easily Share Piles of Code
Here is my code for the dataset: CodePile | Easily Share Piles of Code
How can i achieve what i want?
Thanks in advance.

Context:
in your code , it’s for creating val and train set .
this is one way of spliting your dataset to train and val set.
if you set sampler to None, dataloader choose samples from all semple in your dataset.

 muhammedcanpirincci:

i want my program to select batch that doesnt have augmented image.

transform doesn’t  work this way by default . in most case dataset getitem applis transform to  single sample.
if you want , you can return both transfomed and orginal image in getitem and use one that you want.
def __getitem__(self, i):
        img , t = ...#load image and label
        return img ,self.transform(img),t


How to shuffle multi-worker IterableDataset loading

Question:
My data is stored in class-specific files, each file contains all the data for one class.
I’m currently loading it with a custom IterableDataset and a multi-worker DataLoader. As my dataset is quite large, each worker is responsible for a fixed set of n_files/n_workers files as I do not want to every file into memory on every worker.
The problem is as each file is as class-specific, each worker only produces a batch containing the classes it has been assigned (based on worker ID). Each worker has it’s own copy of dataset and collate_fn so it batches within the worker.
How to shuffle an iterable dataset 1 discusses how to shuffle using torch.utils.data.datapipes.iter.combinatorics.ShuffleIterDataPipe (which isn’t in the docs?). It applies to any iterator but applying it to Dataset then shuffles within each worker and thus batch or to the DataLoader which then only shuffles the order the batches are yielded. The shuffle flag in the DataLoader also just throws me an error about not selecting mode.
Is there some argument in DataLoader that can let me mix and re-collect some buffer of completed batches? Ideally I don’t want to just iterate and collect some n_worker batches and manually shuffle them.
Thanks!
Here’s a simple example of what I’m doing
class CustomDataset(IterableDataset):
    def __init__(self, files: List[str]):
        self.files = files

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()

        if worker_info is None:
            files_chunk = self.files
        else:
            n_workers = worker_info.num_workers
            n_files = len(self.files)
            chunk_size = n_files // n_workers

            chunk_start = chunk_size * worker_info.id
            files_chunk = self.files[chunk_start: chunk_start + chunk_size]

        files_data = [open(f, 'r') for f in files_chunk]
        for line in chain.from_iterable(zip(*files_data)):
            yield line

    dataloader = DataLoader(dataset,
                            batch_size=args.batch_size,
                            num_workers=n_cpus,
                            pin_memory=True)

Context:
I’ve managed to find a solution that works, even if it is a bit ugly.
  dataloader = DataLoader(dataset,
                          batch_size=1,
                          num_workers=n_cpus,
                          collate_fn=lambda batch: {k:v[0] for k,v in default_collate(batch).items()},
                          prefetch_factor=1)
  shuffled = combinatorics.ShufflerIterDataPipe(dataloader, buffer_size=2 * args.batch_size)
  dataloader = DataLoader(shuffled,
                          batch_size=args.batch_size,
                          num_workers=0,
                          collate_fn=custom_collate_fn)

The lambda is to reset the extra dimension added to the front when the first dataloader runs with a batch_size=1. ShufflerIterDataPipe then shuffles the single instances before they are batched by the second data loader. In this case, by running it with num_workers=0 I can also run GPU operations in there, though I’ve found it to conflict with pin_memory=True.


DataLoader multiple workers works for torchvision dataSet but not my custom dataset

Question:
Hi!
First off all, I am reading posts and github issues and threads since a few hours. I learned that Multithreading on Windows and/or Jupyter (Google colab) seams to be a pain or not working at all.
After a lot of trial and error,  following a lot of advice it seams to work now for me, giving me an immense speed improvement. But sadly only with a downloaded Dataset. If I try it with my own it freezes up immidiatly and I have to restart the runtime, all meanwhile the CPU sits at 0%
I also read a lot of posts that are similiar to this one, and none seam to have the same issue. I believe there is an error in my DataSet class that I am unable to find. I hope someone can help me out.
This Test works perfectly fine:
import time
from tqdm import tqdm
import torch
import torchvision

def train(data_loader):
    start = time.time()
    for _ in tqdm(range(10)):
        for x in data_loader:
            pass
    end = time.time()
    return end - start

if __name__ == '__main__':
    train_dataset = torchvision.datasets.FashionMNIST(
        root=".", train=True, download=True,
        transform=torchvision.transforms.ToTensor()
    )

    batch_size = 32
    train_loader1 = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=0)
    train_loader2 = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=8)
    train_loader3 = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=8, persistent_workers=True)
    train_loader4 = torch.utils.data.DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=10, persistent_workers=True)

    print(train(train_loader1))
    print(train(train_loader2))
    print(train(train_loader3))
    print(train(train_loader4))

this gets stuck immediatly after data_loader1, so as soon as I switch on multiple workers. It freezes and the CPU chills at 0%.
It looks like the iterator just never returnes a value? Similar to Multiple Dataloader Workers in multi Threading 1 ?
import time
from torch.utils.data import DataLoader  # Gives easier dataset managment by creating mini batches etc.
from tqdm import tqdm  # For nice progress bar!
import numpy as np

def train(data_loader):
    start = time.time()
    for _ in tqdm(range(10)):
        for x in data_loader:
            pass
    end = time.time()
    return end - start

if __name__ ==  '__main__':
  batch_size = 64

  root_dir = R'[REDACTED]\MiniStoneShaderDataset'
  dataset = smallShaderDataset(csv_file='Labels.csv', root_dir=root_dir )
  train_dataset, test_dataset = torch.utils.data.random_split(dataset, [99, 9900])

  train_loader1 = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=0, pin_memory= True)
  train_loader2 = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=4, pin_memory= True, persistent_workers = True)
  train_loader3 = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, num_workers=8, pin_memory= True, persistent_workers = True)

  print(train(train_loader1))
  print(train(train_loader2))
  print(train(train_loader3))

Using this DataSet Class:
import pandas as pd
import torch
from torch.utils.data import Dataset
import numpy as np

#from skimage import io

class smallShaderDataset(Dataset):
  def __init__(self, csv_file, root_dir, transform=None):
    self.annotations = pd.read_csv(os.path.join(root_dir,csv_file))
    self.root_dir = root_dir
    self.transform = transform

  def __len__(self):
    return len(self.annotations)

  def __getitem__(self, index):

    data_path = os.path.join(self.root_dir,'LearnDataCombined', self.annotations.iloc[index,0] + ".npy")
    input = np.load(data_path)

    parameters = self.annotations.iloc[index,1:]
    parameters = np.array([parameters],dtype = float).flatten()
    input = np.array(input,dtype = float)

    sample = {'input': input, 'parameters': parameters}

    return input, parameters
    #return sample

Everything works (slowly) with num_workers = 0
There is probably  something wrong here wich i am unable to find in full tunnelvision mode. I really hope someone can help me find the issue.
If it really is a Bug in DataSet, why does the downloaded set work fine?
Thank you for any help!
Greetings

Context:
def train(data_loader):
    start = time.time()
    for _ in tqdm(range(10)):
        for x in data_loader:
            pass
    end = time.time()
    return end - start

This part is super confusing. Is this intentional that you want to iterate over DataLoader 10 times?


Pytorch DataLoader re-imports its own caller

Question:
I am using torch 1.9.0 in Python 3.7.6 on Windows 10.
I am finding that each DataLoader worker, when starting up, imports the script from which the DataLoader is being called. I tried to make a minimal example below.
Is this the expected behavior? It really surprises me, since it seems to assume the caller script can be re-imported with no side-effects in every case. Or am I using DataLoader incorrectly?

import inspect
import torchvision
import torch.utils.data as tu_data

def get_dl():
    batch_size = 1
    num_workers = 4

    print('batch_size:', batch_size)
    print('num_workers:', num_workers)

    transform=torchvision.transforms.ToTensor()

    mnist_data = torchvision.datasets.MNIST('./data/',
                                          transform=transform, download=True)
    data_loader = tu_data.DataLoader(mnist_data,
                                          batch_size=batch_size,
                                          shuffle=False,
                                          num_workers=num_workers,
                                          pin_memory=True)

    count = 0
    print('Entering data_loader loop....')
    for datum in data_loader:
        print('count:',count)
        count+=1
        if count > 5:
            break

if __name__ == '__main__':
    get_dl()
else:
    print('MYSTERY IMPORT!', flush=True)
    print('__name__ is',__name__)
    print('Importer is:', inspect.currentframe().f_back.f_code.co_name) # Print out how we got here.

The output looks like this (I am not worried about the warning):
C:\Users\peria\Anaconda3\envs\segmenter\lib\site-packages\torchvision\datasets\mnist.py:498: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  ..\torch\csrc\utils\tensor_numpy.cpp:180.)
  return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)
Entering data_loader loop....
MYSTERY IMPORT!
__name__ is __mp_main__
Importer is: _run_code
MYSTERY IMPORT!
__name__ is __mp_main__
Importer is: _run_code
MYSTERY IMPORT!
__name__ is __mp_main__
Importer is: _run_code
MYSTERY IMPORT!
__name__ is __mp_main__
Importer is: _run_code
count: 0
count: 1
count: 2
count: 3
count: 4
count: 5

Context:
Yes, I think this is expected as Windows uses spawn instead of fork as described in the Windows FAQ 1:

The implementation of multiprocessing is different on Windows, which uses spawn instead of fork. So we have to wrap the code with an if-clause to protect the code from executing multiple times. Refactor your code into the following structure.

import torch

def main()
    for i, data in enumerate(dataloader):
        # do something here

if __name__ == '__main__':
    main()


Appending a tensor of high dimension to create training set

Question:
Hi,
I have an image of dimension 1024x1024 which after flatteining comes out to be a 1048576x1 .
Apart from this I have a json file containing the coordinate information(3x1).In all there are 17,000 entries in that json file.
As such,I am appending the 3x1 vector to the flattened image ,this will create one sample of dimension 1048579x1
The final dataset dimension will be of 1048579x17,000.
Is there an efficient way to create this and load it appropriately in pytorch
I tried reducing the size of the image to 256x256,but my current code leads to a memory issue(even when it’s run on colab pro).
My current code (which was written in numpy is as follows)
import json
import cv2
import numpy as np

image=cv2.imread('blender_files/Model_0/image_0/model_0_0.jpg',0)
shape=image.shape
image_1=cv2.resize(image,(256,256))
flat_img=image_1.ravel()
print(flat_img.shape)
with open('blender_files/Model_0/image_0/vertices_0_0.json') as f:
    json1_str = f.read()
    json1_data=json.loads(json1_str)
print(len(json1_data))
local_coordinates=[]

for i in range(len(json1_data)):
    local_coord=json1_data[i]['local_coordinate']
    local_coord=np.array(local_coord)
    new_arr=np.hstack((flat_img,local_coord))
    new_arr=new_arr.tolist()
    local_coordinates.append(new_arr)
#local_coords=np.array(json1_data['local_coordinate'])
#print(local_coords.shape)
#new_arr=np.hstack((flat_img,local_coords))
print(new_arr.shape)
local_coordinates=np.array(local_coordinates)
print(local_coordinates.shape)

Context:
In my experience, it is not good approach to stack a such large tensor at once.
I recommend you to store (256x256 + 3, 1) size of tensor which indicates a single data.
Storing each data points separately has an advantage of easy managing and fast loading and so forth.
Think about why such a huge datasets (well-known open datasets) manages all data points(images) separately.


How to split your training data into indexable batches?

Question:
I’m trying to manually split my training data into individual batches in a way where I can access the desired batch by indexing. Hence, I can’t rely on DataLoader to do the batch splitting since it’s unindexable. I’ve tried some approaches to achieve this as per this link 2, but I’ve been getting some weird behavior.
So what is the proper way to implement this?

Context:
@Omar_AlSuwaidi , you should be able to use a simple list for achieving the indexer
import math
import torch
import torch.nn as nn

X = torch.rand(1000,10, 4)
batch_size = 64
num_batches = math.ceil(X.size()[0]/batch_size)
X_list = [X[batch_size*y:batch_size*(y+1),:,:] for y in range(num_batches)]
print(X_list[0].size())

torch.Size([64, 10, 4])