Getting closer. project and unproject based on the current zoom seems to work in most cases.