[library] Don't handle touch events when adapter is null

Fixes #19