Simplify number handling to just process as double

I am not entirely sure if there is any point in detecting if the
number is actually a double or an int. The overhead of detecting
which to use may outweigh that of using it. Will have to test
further, but if we want to go back to using the double/int
determination, then revert this commit