It seems like the hand pose need to be replaced after initial contact so the motion is repeatable.