Custody changes or modifications are simple if both parents are in agreement. If parents are not in agreement, such as one parent thinks the other is unfit, it is much more challenging. For the most part, most states want both parents to help raise the children, even if things don't go quite right. In most cases, supervised visitation would be enforced before a court-ordered custody modification.
However, modification to custody agreements may be considered by the courts in a few situations, such as:
Domestic violence or child abuse
Parent relocates, necessitating the need to modify the original agreement
A parent routinely doesn't follow the visitation schedule
A custodial parent dies
Custodial parent's living situation is unsafe and/or unsanitary
The child is failing in school
Courts are not usually quick to change custody and will consider the best interest of the child before parental interests. You may need to hire a lawyer or seek social services for support.