Repository: JuliaGPU/CUDA.jl

Issue #1314: Segfault with CUDA in a sysimage
Created: 2022-01-10T19:48:16Z
Workaround: take CUDA.jl out of the sysimage 😄. Also seems to work on Julia 1.6.5 (but not 1.7.1).
Cause: during sysimage creation we create broken code, according to @vchuravy who helped me try to debug a segfault on #gpu on Slack. He pointed to

  
    
      CUDA.jl/lib/cudadrv/execution.jl
    
    
         Line 143
      in
      3497077
    
  
  
    

        
          
           async_send(data::Ptr{Cvoid}) = ccall(:uv_async_send, Cint, (Ptr{Cvoid},), data) 
        
    
  


and
julia> @code_llvm CUDA.async_send(C_NULL)
;  @ /root/.julia/packages/CUDA/Rhl18/lib/cudadrv/execution.jl:143 within `async_send`
define i32 @julia_async_send_1861(i64 zeroext %0) #0 {
top:
  %1 = call i32 inttoptr (i64 139633100932974 to i32 (i64)*)(i64 %0)
  ret i32 %1
}

and said

You are reading from TLS on a foreign thread which is illegal

I don't understand most of that to be honest.
This is with CUDA#master (or v3.6.1), Julia 1.7.1, PackageCompiler v1.7.7.

Debugger session:
[New Thread 0x7f8ae0a22700 (LWP 660)]
[Thread 0x7f8ae0a22700 (LWP 660) exited]
[New Thread 0x7f8ae0a22700 (LWP 661)]

Thread 20 "julia" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f8ae0a22700 (LWP 661)]
0x00007f8bab618905 in jlcapi_async_send_69899 () from /deps.so
(gdb) bt
#0  0x00007f8bab618905 in jlcapi_async_send_69899 () from /deps.so
#1  0x00007f8b30e2045a in ?? ()
   from /root/.julia/artifacts/b618eadab05abb3bef21da981a51b6f4f0183b27/lib/libcuda.so
#2  0x00007f8b30e88106 in ?? ()
   from /root/.julia/artifacts/b618eadab05abb3bef21da981a51b6f4f0183b27/lib/libcuda.so
#3  0x00007f8bd5628609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#4  0x00007f8bd554f293 in clone () from /usr/lib/x86_64-linux-gnu/libc.so.6
(gdb) disassemble jlcapi_async_send_69899
Dump of assembler code for function jlcapi_async_send_69899:
   0x00007f8bab6188c0 <+0>:     push   %r15
   0x00007f8bab6188c2 <+2>:     push   %r14
   0x00007f8bab6188c4 <+4>:     push   %r13
   0x00007f8bab6188c6 <+6>:     push   %r12
   0x00007f8bab6188c8 <+8>:     push   %rbx
   0x00007f8bab6188c9 <+9>:     sub    $0x20,%rsp
   0x00007f8bab6188cd <+13>:    mov    %rdi,%r14
   0x00007f8bab6188d0 <+16>:    xorps  %xmm0,%xmm0
   0x00007f8bab6188d3 <+19>:    movaps %xmm0,(%rsp)
   0x00007f8bab6188d7 <+23>:    movq   $0x0,0x10(%rsp)
   0x00007f8bab6188e0 <+32>:    mov    0x1e902459(%rip),%rax        # 0x7f8bc9f1ad40 <jl_tls_offset.real>
   0x00007f8bab6188e7 <+39>:    test   %rax,%rax
   0x00007f8bab6188ea <+42>:    je     0x7f8bab618999 <jlcapi_async_send_69899+217>
   0x00007f8bab6188f0 <+48>:    mov    %fs:0x0,%rcx
   0x00007f8bab6188f9 <+57>:    mov    (%rcx,%rax,1),%rbx
   0x00007f8bab6188fd <+61>:    movq   $0x4,(%rsp)
=> 0x00007f8bab618905 <+69>:    mov    (%rbx),%rax
--Type <RET> for more, q to quit, c to continue without paging--

Duplicate issue #1062: Segfault during CUBLAS logging
Created: 2021-07-22T16:12:56Z
Apparently there's still some issue with the logger:

As reported by @femtomc, encountered on CUDA.jl 3.3.4 with JULIA_DEBUG=CUDA.

Closing comment (2022-01-11T20:53:50Z): @femtomc pointed out this could be related to #1314 (I believe the issue did occur with CUDA in a sysimage)