Push unique_ptr a bit further through some APIs and simplify some cleanup