If you have the power of attorney, the "hired help" cannot legally exclude you if you have a broad-based POA; if you have  broad POA, you most likely can also fire these people if you like. You are advised to speak with an elder law attorney who can review the POA with you to determine your exact rights and power, and also advise you as to how to handle your grandmother seeming to be taken advantage of--it may be that a lawsuit, or even criminal charges, against these people is appropriate.