We felt unable to give this more than three stars because of the limited access: during our visit the staircase to Oia was under repair (part of it has fallen away in a rockslide). The only official access to Oia is via a 5-minute boat transfer for which we were asked to pay 20 euros each way. That's 40 euros extra per day to leave the villa once. We were only told about this a few days before our arrival and after the cancellation period, so it left us feeling very cheated. We used the stairs anyway but you do need to be careful.
The villa itself is small and has too much furniture so it feels cramped. Neither of us felt the ladder to the top sleeping spaces was secure enough to use at night. Also due to poor ventilation there are small patches of mould starting to grow on the ceilings. In addition there is no WiFi (it's not falsely advertised in the listing but something you might expect). It might be okay for two nights but it wasn't great for our visit of six nights.
On the plus side, Armeni Bay is beautiful and peaceful, the views are stunning and the restaurant next door is lovely.
Overall we were disappointed given the extremely positive reviews we had seen from previous guests.