Fix minor UI issue

When aborting a swipe-from-left gesture very fast, it was possible to
get the navigation bar detached from its normal position.