Yes, even though the chicken is cooking in a liquid, it can still dry out if cooked for too long at too high of a temperature. Once the chicken is cooked through, turn the soup right down to a low heat to prevent overcooking the chicken.