Revert "manager: Don't reuse the same X server when we have a display server"

This reverts commit 72ccb27377886b9998ad8b5b7a3b3070474f90f3.

We're going to take a different approach for this.