Assign empty shared_ptr instead of extra remove function.