Prevent scroll drifting in swipe recovery

Change-Id: I8179607517f55b2941924b5726e0e57b012f8e18