Yes, it is very easy to overcook and dry out chicken on the smoker! I prefer to use bone-in, skin-on chicken thighs for this reason because they retain moisture better than the skinless thighs. To avoid overcooking your meat, make sure to use a meat thermometer and remove the chicken as soon as it reaches the correct internal temperature.