Remove casts where possible

This PR removes casts where possible. Please review _carefully_, I did no
double-checking whether I accidentially changed functionality!

Compiled and tested (`make test`) sucessfully!