Put reply/enqueue outside of the lock and use shared_ptr instead of iterator->shared_ptr