The main setbacks include potential quality issues, loss of control over processes, communication challenges, impact on employee morale, and complications in ending the relationship.