I tried to run caffe with large net OverFeat ( http://cilvr.nyu.edu/doku.php?id=software:overfeat:start ) on K20, but model did not fit K20 DRAM. I was able to train it on CPU. The footprint for 16 threads was ~ 16 GB(!). The train speed was ~ 5 images/sec. I used server with 32 GB , 2x E5x2670 2.6 GHz.