RuntimeError: CUDA error: an illegal memory access was encountered, torch/cuda/streams.py", line 94, in synchronize
Labels: module: cuda, triaged

  Hi.     
I get a torch.cuda.synchronize() error, when i infer the TRT plugin. The detailed error information is as follows.


        Building TRT engine....
        [04/03/2023-06:52:48] [TRT] [V] Applying generic optimizations to the graph for inference.
        [04/03/2023-06:52:48] [TRT] [V] Original: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After dead-layer removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After Myelin optimization: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] Applying ScaleNodes fusions.
        [04/03/2023-06:52:48] [TRT] [V] After scale fusion: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After dupe layer removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After final dead-layer removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After tensor merging: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After vertical fusions: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After dupe layer removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After final dead-layer removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After tensor merging: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After slice removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] After concat removal: 1 layers
        [04/03/2023-06:52:48] [TRT] [V] Trying to split Reshape and strided tensor
        [04/03/2023-06:52:48] [TRT] [V] Graph construction and optimization completed in 0.109626 seconds.
        [04/03/2023-06:52:49] [TRT] [V] Using cublasLt as a tactic source
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +67, GPU +8, now: CPU 4520, GPU 3365 (MiB)
        [04/03/2023-06:52:49] [TRT] [V] Using cuDNN as a tactic source
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 4520, GPU 3375 (MiB)
        [04/03/2023-06:52:49] [TRT] [I] Local timing cache in use. Profiling results in this builder pass will not be stored.
        [04/03/2023-06:52:49] [TRT] [V] Constructing optimization profile number 0 [1/1].
        [04/03/2023-06:52:49] [TRT] [V] Reserving memory for host IO tensors. Host: 0 bytes
        [04/03/2023-06:52:49] [TRT] [V] =============== Computing reformatting costs
        [04/03/2023-06:52:49] [TRT] [V] =============== Computing reformatting costs
        [04/03/2023-06:52:49] [TRT] [V] =============== Computing costs for 
        [04/03/2023-06:52:49] [TRT] [V] *************** Autotuning format combination: Float(1327104,20736,144,1) -> Float(165888,512,1) ***************
        [04/03/2023-06:52:49] [TRT] [V] Formats and tactics selection completed in 0.0748828 seconds.
        [04/03/2023-06:52:49] [TRT] [V] After reformat layers: 1 layers
        [04/03/2023-06:52:49] [TRT] [V] Pre-optimized block assignment.
        [04/03/2023-06:52:49] [TRT] [V] Block size 8589934592
        [04/03/2023-06:52:49] [TRT] [V] Total Activation Memory: 8589934592
        [04/03/2023-06:52:49] [TRT] [I] Detected 1 inputs and 1 output network tensors.
        [04/03/2023-06:52:49] [TRT] [V] Layer: (Unnamed Layer* 0) [PluginV2DynamicExt] Host Persistent: 112 Device Persistent: 0 Scratch Memory: 0
        [04/03/2023-06:52:49] [TRT] [I] Total Host Persistent Memory: 112
        [04/03/2023-06:52:49] [TRT] [I] Total Device Persistent Memory: 0
        [04/03/2023-06:52:49] [TRT] [I] Total Scratch Memory: 0
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 0 MiB
        [04/03/2023-06:52:49] [TRT] [V] Optimized block assignment.
        [04/03/2023-06:52:49] [TRT] [I] Total Activation Memory: 0
        [04/03/2023-06:52:49] [TRT] [V] Disabling unused tactic source: EDGE_MASK_CONVOLUTIONS
        [04/03/2023-06:52:49] [TRT] [V] Using cublasLt as a tactic source
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +1, GPU +8, now: CPU 4619, GPU 3821 (MiB)
        [04/03/2023-06:52:49] [TRT] [V] Using cuDNN as a tactic source
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 4619, GPU 3829 (MiB)
        [04/03/2023-06:52:49] [TRT] [V] Engine generation completed in 0.842963 seconds.
        [04/03/2023-06:52:49] [TRT] [V] Engine Layer Information:
        Layer(PluginV2): (Unnamed Layer* 0) [PluginV2DynamicExt], Tactic: 0x0000000000000000, input_img[Float(-2,64,144,144)] -> vision_transformer_output[Float(-2,324,512)]
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +0, now: CPU 0, GPU 0 (MiB)
        [04/03/2023-06:52:49] [TRT] [V] Using cublasLt as a tactic source
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 4573, GPU 3681 (MiB)
        [04/03/2023-06:52:49] [TRT] [V] Using cuDNN as a tactic source
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 4573, GPU 3689 (MiB)
        [04/03/2023-06:52:49] [TRT] [V] Total per-runner device persistent memory is 0
        [04/03/2023-06:52:49] [TRT] [V] Total per-runner host persistent memory is 112
        [04/03/2023-06:52:49] [TRT] [V] Allocated activation device memory of size 0
        [04/03/2023-06:52:49] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 0 (MiB)
     
      Traceback (most recent call last):
        File "examples/infer_visiontransformer_plugin.py", line 310, in <module>
          main(args)
        File "examples/infer_visiontransformer_plugin.py", line 128, in main
          plugin_output = run_trt_plugin(p_loader, images_tensor, engine)
        File "/opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
          return func(*args, **kwargs)
        File "examples/infer_visiontransformer_plugin.py", line 166, in run_trt_plugin
          stream.synchronize()
        File "/opt/conda/lib/python3.8/site-packages/torch/cuda/streams.py", line 94, in synchronize
          super(Stream, self).synchronize()
      RuntimeError: CUDA error: an illegal memory access was encountered
      CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
      For debugging consider passing CUDA_LAUNCH_BLOCKING=1.


- PyTorch or Caffe2:
- How you installed PyTorch (conda, pip, source):
- Build command you used (if compiling from source):
- OS:
- PyTorch version:
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- GCC version (if compiling from source):
- CMake version:
- Versions of any other relevant libraries:


cc @ngimel