opened
lhoste-bell
username_0
We observe a segfault when running the nvidia-docker-plugin. This is the output:
```
root@serflex-giant-2:~# nvidia-docker-plugin
nvidia-docker-plugin | 2017/06/19 16:36:45 Loading NVIDIA unified memory
nvidia-docker-plugin | 2017/06/19 16:36:45 Loading NVIDIA management library
nvidia-docker-plugin | 2017/06/19 16:36:46 Discovering GPU devices
fatal error: unexpected signal during runtime execution
[signal 0xb code=0x1 addr=0x10 pc=0x7f2a6c2b1d44]

runtime stack:
runtime.throw(0x9398a0, 0x2a)
	/usr/local/go/src/runtime/panic.go:527 +0x90
runtime.sigpanic()
	/usr/local/go/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x710be1, 0xc820071a90, 0x0)
	/usr/local/go/src/runtime/cgocall.go:120 +0x11b fp=0xc820071a60 sp=0xc820071a30
github.com/NVIDIA/nvidia-docker/src/cuda._Cfunc_cudaDeviceGetByPCIBusId(0xc82000b300, 0x28a8080, 0x0)
	??:0 +0x35 fp=0xc820071a90 sp=0xc820071a60
github.com/NVIDIA/nvidia-docker/src/cuda.deviceGetByPCIBusId(0xc82000b1f0, 0xc, 0xc800000000, 0x0, 0x0)
	/go/src/github.com/NVIDIA/nvidia-docker/src/cuda/bindings.go:49 +0x78 fp=0xc820071ac8 sp=0xc820071a90
github.com/NVIDIA/nvidia-docker/src/cuda.NewDevice(0xc82000b1f0, 0xc, 0x0, 0x0, 0x0)
	/go/src/github.com/NVIDIA/nvidia-docker/src/cuda/cuda.go:76 +0x54 fp=0xc820071bf0 sp=0xc820071ac8
github.com/NVIDIA/nvidia-docker/src/nvidia.LookupDevices(0x0, 0x0, 0x0, 0xc82000eb40, 0x0, 0x2, 0x0, 0x0)
	/go/src/github.com/NVIDIA/nvidia-docker/src/nvidia/devices.go:67 +0x34b fp=0xc820071cf0 sp=0xc820071bf0
main.main()
	/go/src/github.com/NVIDIA/nvidia-docker/src/nvidia-docker-plugin/main.go:74 +0x4ab fp=0xc820071f50 sp=0xc820071cf0
runtime.main()
	/usr/local/go/src/runtime/proc.go:111 +0x2b0 fp=0xc820071fa0 sp=0xc820071f50
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc820071fa8 sp=0xc820071fa0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 5 [syscall]:
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x37
```

Our nvidia-smi (native) provides the following configuration:
```
root@serflex-giant-2:~# nvidia-smi
Mon Jun 19 16:37:11 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 980     Off  | 0000:02:00.0     Off |                  N/A |
| 26%   36C    P8    11W / 180W |    107MiB /  4038MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 980     Off  | 0000:81:00.0     Off |                  N/A |
| 26%   33C    P8    10W / 180W |    107MiB /  4038MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      4239    C   nvidia-cuda-mps-server                          96MiB |
|    1      4239    C   nvidia-cuda-mps-server                          96MiB |
+-----------------------------------------------------------------------------+
```
```
root@serflex-giant-2:~# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17
```

We've rebuild nvidia-docker with Golang 1.7 but that triggered the same segfault. Any ideas?
SIGSEGV src/cuda.deviceGetByPCIBusId
issue
created
flx42
username_1
Do you need CUDA MPS in your setup? If not, you should kill `nvidia-cuda-mps-server` and try again.
comment
closed
lhoste-bell
username_0
issue
created
lhoste-bell
username_0
That fixed it, thanks.
comment