Fix: Don't move the handle if not touch it