Theano/Theano — Issue #507: gpu local fusion and memory error
Author: jaberg
Created: 2012-03-06 23:38:20+00:00
State: open
Labels: Bug, High Priority

I'm trying to diagnose a GPU memory-error (launch failure, failure to free pointer, etc.) in a large model, which occurs only on some machines. It's consistently reproducible on some machines. It does not seem related to the particular theano version (no difference between 0.5rc2, 0.5, or HEAD). The graph is a feedforward convolutional model with lots of elemwise, subtensors with negative strides, valid-mode convolutions, large constants, and reductions. No updates. I'll be trying to narrow things down by triggering the crash with smaller graphs, but in the meantime I just wanted to give a heads up that something's amiss, and ask if anyone else has seen similar things recently, or has any information regarding this bug, to contact the authorities, er, mailing list.

The most interesting hints I have gleaned is that the crash (on those machines which crash this way)
- occurs on the first run through the linker,
- occurs only for certain array sizes
- can be avoided by setting config.gpu.local_gpu_fusion=0.

Hopefully I'll get it sorted tomorrow and be able to provide more info.