The attorney-in-fact is legally obligated to act in the best interests of the principal. If a close relative, spouse, close friend, or someone else believes that they do not fulfill their promise, they may try to override the power of attorney.
That said, this can only be done by litigation. The challenger should prove that the attorney-in-fact is misusing their power and then the court will decide the status of the power of attorney. Since these proceedings may take months before hearing, the principal should only choose someone who will act in their best interest as an agent.