Remove a bunch of superfluous casts

They just ugly up the code and could potentially hide errors.

The only problem it might cause is if using C++ compiler and,
well, don't do that. This is C code, use the right compiler :)