Only the principal who issued the power of attorney can change their attorney-in-fact. They can revoke PoA or sign another PoA, informing their former agent their services are no longer needed.
A third party like a loved one or a family member can only challenge the validity of the legal power of attorney in court.