The person who has made the power of attorney can cancel an LPA at any time - they don't need to give a reason. The only reason this might be complicated is if the person that made the LPA no longer has mental capacity.
It is also possible to challenge an LPA by filing a petition in court to challenge the attorney if their attorney isn’t acting in their best interest. If successful the court will cancel the LPA and appoint an alternative deputy.