Update to latest Rust

This updates the code to the latest Rust. The highlights are:

* Removes old borrow checker work arounds
* Appeases the new borrow checker
* Removes most warnings (unused unsafe and unused mut mostly)