Restrict /all the pointers/

This enables various optimisations that can't be done when pointers
alias.