Fix removing scroll listeners

When done function (_endLoadMore) is called faster (e.g. no loading is
actually needed) than _removeListeners after loadMore, all listeners are
removed. This is due _addListeners is called prior _removeListeners