Remove the remaining raw pointers

Fixes #195
I'll leave that for another issue