Fix incorrect scrolling in the mini map, even when no button is pressed