Fix type of - to not be wrong on negative fixnums

Rolling this forward because the math library is no longer dependent on
the broken behavior. Also nothing in the main repo seems to depend on
this.