Restructured code to allow a separate Display.

The point of that is to make sure we don't interfere with the
X window that's past to us. We shouldn't rely or mess with
the ownership of events of another window.