Fix: Don't move the handle if it isn't touched