I think you should limit the maximum size of shared memory used. If I ramp up the number of columns in the dense matrix to e.g. 3000, the GPU is easily out of memory.