No, once the robber is out of the desert they cannot go back. You are free to move it to a blank space if you don't want to be mean, but that is about it.