Also set swipe direction when setting position