[report a bug] I allocated some GPU memory using Torch, and I want to copy data from the CPU to the GPU, but it hasn’t been successful

username: Jiawen_Niu
cooked: <p>Here is a simplified program that doesn’t work properly and throws the following error:</p>
<pre><code class="lang-auto">
#include &lt;torch/torch.h&gt;
#include &lt;torch/csrc/distributed/c10d/ProcessGroupNCCL.hpp&gt;
#include &lt;torch/csrc/distributed/c10d/FileStore.hpp&gt;
#include &lt;c10/cuda/CUDAStream.h&gt; // For CUDAStreamGuard and getStreamFromPool
#include &lt;c10/cuda/CUDAFunctions.h&gt; // For CUDASetDevice, device_count, is_available
#include &lt;c10/core/DeviceGuard.h&gt; // For CUDAGuard
#include &lt;cstdlib&gt; // for std::getenv, std::stoi
#include &lt;iostream&gt;
#include &lt;string&gt;
#include &lt;vector&gt;
#include &lt;stdexcept&gt; // For std::runtime_error, std::exception
#include &lt;memory&gt; // For c10::make_intrusive
#include &lt;chrono&gt; // For timeout duration and sleep_for
#include &lt;thread&gt; // For std::this_thread::sleep_for
#include &lt;cstdio&gt; // For std::remove

int main(int argc, char* argv[]) {
    // Define the size of memory to allocate (e.g., 1KB)
    size_t size_bytes =  1024; // 1 KB

    // 1. Prepare data on the CPU
    std::vector&lt;char&gt; cpu_data(size_bytes);
    for (size_t i = 0; i &lt; size_bytes; ++i) {
        cpu_data[i] = static_cast&lt;char&gt;(i % 128);
    }
    char* cpu_ptr = cpu_data.data();
    std::cout &lt;&lt; "CPU data prepared, size: " &lt;&lt; size_bytes &lt;&lt; " bytes." &lt;&lt; std::endl;

    // Check if CUDA is available
    if (!torch::cuda::is_available()) {
        std::cerr &lt;&lt; "Error: CUDA is not available. This test requires CUDA." &lt;&lt; std::endl;
        return 1; // Exit if CUDA is not available
    } else {
        int device_count = torch::cuda::device_count();
        std::cout &lt;&lt; "CUDA is available. Found " &lt;&lt; device_count &lt;&lt; " GPU device(s)." &lt;&lt; std::endl;
        if (device_count == 0) {
             std::cerr &lt;&lt; "Error: CUDA is available but no GPU devices were found." &lt;&lt; std::endl;
             return 1;
        }
    }

    void* gpu_ptr = nullptr;
    try {
        // (Optional) Set the CUDA device to operate on, default is 0
        c10::cuda::set_device(0);
        c10::DeviceGuard device_guard(c10::Device(c10::kCUDA, 0)); // Ensure subsequent operations are on the specified device

        // 2. Allocate GPU memory using CUDACachingAllocator
        std::cout &lt;&lt; "Attempting to allocate " &lt;&lt; size_bytes &lt;&lt; " bytes of GPU memory..." &lt;&lt; std::endl;
        gpu_ptr = c10::cuda::CUDACachingAllocator::raw_alloc(size_bytes);
        std::cout &lt;&lt; "Attempting to copy CPU data to GPU..." &lt;&lt; std::endl;
        cudaError_t cuda_err = cudaMemcpy(gpu_ptr, cpu_ptr, size_bytes, cudaMemcpyHostToDevice);
        if (cuda_err != cudaSuccess) {
            std::cerr &lt;&lt; "Error: cudaMemcpy HostToDevice failed: " &lt;&lt; cudaGetErrorString(cuda_err) &lt;&lt; std::endl;
            // Free allocated GPU memory on error
            c10::cuda::CUDACachingAllocator::raw_delete(gpu_ptr);
            return 1;
        }
        std::cout &lt;&lt; "Data successfully copied from CPU to GPU." &lt;&lt; std::endl;

        // (Optional) Add code here to use the GPU memory pointed to by gpu_ptr...

        // 4. Free GPU memory
        std::cout &lt;&lt; "Freeing GPU memory..." &lt;&lt; std::endl;
        c10::cuda::CUDACachingAllocator::raw_delete(gpu_ptr);
        gpu_ptr = nullptr; // Set pointer to null to avoid dangling pointer
        std::cout &lt;&lt; "GPU memory has been freed." &lt;&lt; std::endl;

    } catch (const c10::CUDAError&amp; e) {
        std::cerr &lt;&lt; "Caught c10::CUDAError: " &lt;&lt; e.what() &lt;&lt; std::endl;
        // In case of exception, try to free memory if it was allocated
        if (gpu_ptr) {
            std::cout &lt;&lt; "Freeing GPU memory in exception handler..." &lt;&lt; std::endl;
            c10::cuda::CUDACachingAllocator::raw_delete(gpu_ptr);
        }
        return 1;
    } catch (const std::exception&amp; e) {
        std::cerr &lt;&lt; "Caught std::exception: " &lt;&lt; e.what() &lt;&lt; std::endl;
        // In case of exception, try to free memory if it was allocated
        if (gpu_ptr) {
             std::cout &lt;&lt; "Freeing GPU memory in exception handler..." &lt;&lt; std::endl;
            c10::cuda::CUDACachingAllocator::raw_delete(gpu_ptr);
        }
        return 1;
    }

    std::cout &lt;&lt; "Memory allocation and copy test completed successfully." &lt;&lt; std::endl;
    return 0; // Main function exits normally
}

</code></pre>
<pre><code class="lang-auto">CPU data prepared, size: 1024 bytes.
CUDA is available. Found 7 GPU device(s).
Attempting to allocate 1024 bytes of GPU memory...
Caught std::exception: 0 &lt;= device &amp;&amp; static_cast&lt;size_t&gt;(device) &lt; device_allocator.size() INTERNAL ASSERT FAILED at "../c10/cuda/CUDACachingAllocator.cpp":2906, please report a bug to PyTorch. Allocator not initialized for device
</code></pre>
<p>The version of libtorch I am using is 2.3.1+cu118.</p>

username: Jiawen_Niu
cooked: <p>Before allocating GPU memory, I added a line to allocate a tensor on the GPU, and it started working. It seems that this correctly initializes the device. Why is this happening?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4428dcd0dfbeb1010b62fe46b00ed5a6c928d88.png" data-download-href="https://discuss.pytorch.org/uploads/default/e4428dcd0dfbeb1010b62fe46b00ed5a6c928d88" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4428dcd0dfbeb1010b62fe46b00ed5a6c928d88_2_690x368.png" alt="image" data-base62-sha1="wzhsay320OrDA9Haq4qwf3XXKre" width="690" height="368" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4428dcd0dfbeb1010b62fe46b00ed5a6c928d88_2_690x368.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e4428dcd0dfbeb1010b62fe46b00ed5a6c928d88_2_1035x552.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/4/e4428dcd0dfbeb1010b62fe46b00ed5a6c928d88.png 2x" data-dominant-color="F5F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1284×686 116 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Did you try to initialize the allocator via <code>c10::cuda::CUDACachingAllocator::init</code>?</p>

username: Jiawen_Niu
cooked: <p>This solves the problem, thank you very much for your reply.</p>