[Fix #664] Remove controversial other-window remapping

This also closes #656.