Repository: NVIDIA/cccl

Issue #1474: [FEA]: Not all copies in Thrust support `par_nosync`: only D->D for non-trivial types can be executed without synchronization.
Created: 2024-03-02T23:20:09Z
Is this a duplicate?

 I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct

Type of Bug
Performance
Component
Thrust
Describe the bug
For device-to-device copies using thrust::copy, non-trivial types are routed through thrust::transform, which is implemented using cuda_cub::parallel_for and will not synchronized when used with CUDA's par_nosync execution policy. This is good.
Trivially relocatable types, however, use cudaMemcpyAsync followed by a non-optional synchronization.
We should change the cuda_cub::synchronize(policy) call to cuda_cub::synchronize_optional(policy), after ensuring that this won't break any internal copies (e.g. to/from temp_storage) that may rely on non-optional synchronization.
Trivially relocatable H->D and D->H copies also use the same pattern of cudaMemcpyAsync + sync, but would take additional refactoring to pass the policy into the implementation, instead of just the stream. This will also require some changes to the implementation of cross-system non-trivial copies, which rely on the trivial copy implementations internally and need to always synchronize (D->H in particular).
Copy currently takes a single execution policy that defines the memory space for both input and output ranges, and this poses an additional barrier to using the CUDA par_nosync policy for cross-system copies.
These implementations are located here.
How to Reproduce
Use thrust::copy(thrust::cuda::par_nosync, ...) for anything but a device-to-device copy of non-trivially-relocatable types. The call will synchronize.
Expected behavior
The call should not synchronize.
Reproduction link
No response
Operating System
No response
nvidia-smi output
No response
NVCC version
No response

Duplicate issue #2397: [FEA] Performing device-to-device copies with thrust::copy is synchronous with `par_nosync`. Can this be improved?
Created: 2024-09-10T12:21:35Z
Is this a duplicate?

 I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct

Area
Thrust
Is your feature request related to a problem? Please describe.
The underlying function trivial_copy_device_to_device is always synchronous, as it does not use cuda_cub::synchronize_optional.

  
    
      cccl/thrust/thrust/system/cuda/detail/util.h
    
    
        Lines 165 to 180
      in
      ee9b856
    
  
  
    

        
          
           THRUST_RUNTIME_FUNCTION cudaError_t 
        

        
          
           trivial_copy_device_to_device(Policy& policy, Type* dst, Type const* src, size_t count) 
        

        
          
           { 
        

        
          
             cudaError_t status = cudaSuccess; 
        

        
          
             if (count == 0) 
        

        
          
             { 
        

        
          
               return status; 
        

        
          
             } 
        

        
          
            
        

        
          
             cudaStream_t stream = cuda_cub::stream(policy); 
        

        
          
             // 
        

        
          
             status = ::cudaMemcpyAsync(dst, src, sizeof(Type) * count, cudaMemcpyDeviceToDevice, stream); 
        

        
          
             cuda_cub::synchronize(policy); 
        

        
          
             return status; 
        

        
          
           } 
        

        
          
            
        
    
  


This can also lead to unexpected synchronization in other places, for example thrust::sort will perform a copy internally and thus synchronize.
#include <thrust/sort.h>
#include <thrust/device_vector.h>
#include <thrust/execution_policy.h>

void func(size_t n){
    thrust::device_vector<int> vec(n);
    thrust::sort(thrust::cuda::par_nosync,vec.begin(), vec.end());
}

int main(){
    func(1024*1024);
}

Describe the solution you'd like
D2D copies should be asynchronous with par_nosync
Describe alternatives you've considered
No response
Additional context
At the time of implementing par_nosync (NVIDIA/thrust#1568) there were some places where synchronization was optional in host code, and mandatory in CDP device code and thus left as mandatory in host code as well.
With the changes during the last years , deprecating CDP1 and making thrust algorithms sequential in device code, I wonder if more syncs could be made optional as well.

Closing comment (2024-09-13T06:48:47Z): I just noticed that this issue is a duplicate of #1474