Move some code around and switch over to unique_ptr
Fixes #6