unittests: correct some integers which should have been floating point types

Also works around some weird msvc template / overload resolution issue :/