Yes, in some circumstances an agent may be able to abuse their power. That's why it's important to carefully select who you give POA rights. You also need to consider what powers you are giving them and set out legal documents like Advanced Directives and a Last Will and Testament to ensure that your affairs are handled properly.
The more you decide on now, the less room there is for your agent to cause problems later.