Fix a bug that -4 -> 4294967292 for UInt32

We should cast to more larger signed integer size before "-"
operation.