lib: Prevent starting refresh incorrectly

Only raise the event once per gesture.
Also, if a new gesture is started while refreshing, ignore it.
Removed some unneeded instance variables.