fix #1287  navigation keymap inconsistency

when an escape keystroke is registered before a dialog is opened, the navigation keymap contains 2 objects, and when the dialog is closed, the wrong one is removed.
This fix introduces a comparison between the "modal" property that is used in the dialog keymap configuration, so that the right object is deleted and the user defined one is still handled by its callback.

closes #1287