Stop cloning scroller, just move it around

Also add short-circuit logic to mutation events so we don't always have
to recalc everything.