Finally found the bug. It turns out that the batch size was key - I perform best with just raw gradient descent here (i.e. full batches). Seems SGD is only interesting when the size of the data is really, really large (which is not so much the case here). Also brings forward more opportunities for GPU acceleration once I get to that.