fix subsampling logic

No idea what the original intention was, but it never worked for
g_subsam>=0.501.  This patch makes it work and increases the resolution
somewhat.