Remove unused unsafe blocks, remove #![allow(unused_unsafe)]

I just started looking at the code so I might be missing something. I didn't experience the problem described by the todo this removes, but this seems to work just fine... perhaps the language changed in the mean time.