Assorted small speed optimizations to the drag callback.