Switched to std::shared_ptr, seems to work now