Defer sweep2wake changes till screen is resumed.
Idea taken from some Atmel implementation tho I've lost URI there :(