Fix bug of forgetting to check the target peer is in swarm or not in some speical cases