We really enjoyed to stay in the quiet village of Port de Canonge. For two people the flat was really comfortable, with a special mention for the terrace with great sea view where we've spent most of the time. The little beach of the village is just a 3 minutes away by foot and walking 15 minutes towards the East or West you'll find beautiful "calas" with very few people around. Also, the mountains around are simply beautiful.
You still have to be warned, the road to get there is kind of epic, and the closest place to get groceries is 15-20 minutes away, in Esporles. Also, there are 2 restaurants in Port de Canonge but they close around 5pm.
In short, this is a very good place if you want to be away from the crowds and agitation.