The Caves is a perfect location - beach access, literally 2 mins walk, and 2 great restaurants 5 mins away. also easy to get into town.
however, the inside I'd a bit dark and dated. You're right on top of the other apartments so noise was a problem a couple times. And for light sleepers, the curtains are pretty thin so bright sun in the early morning was a reality!
but if this kind of thing doesn't bother you and if you are just wanting a place with comfy beds and a good shower, and will be spending most of your time out in Byron and at the beach, this place is a good base!