Intrusive pointer -> raw pointer fixes

This fixes up the API to use raw pointers in a few more places where we had been using intrusive pointers unnecessarily.