Remove some unnecessary transmutes.

These can either be done by implicit `&` -> `*` coercions, explicit `*`
-> `*` casts, or an explicit `&*x` `*` -> `&` re-borrow (which is still
unsafe, but significantly more controlled compared to a `transmute`).