Math: fix warning about literal types in normalize().

C++11 changed type of character literal (i.e. '\x7F') from int to char,
thus the documentation is wrong (also mistakenly used octal literal).
Added test to properly test the behavior.