The bug was due to lack of feature scaling: the squared deformation displacements are of a scale much larger than the histogram bins (which are in the [0;1] range), so GD was messing up. Now scaling displacements to the [-1;1] range. This also makes deformation cost much less important to the cost function. There might be a tradeoff to find.