Only remove our only event handler when we destruct the pager

Keep a reference to the scroll callback so that we can remove only our own event handler when we destruct