We (a family of 3 adults + 2 young children) had a wonderful stay at this place over Xmas. It's a short stroll to the Pittwater side, with a very kid-friendly wave-free beach. Shops, cafes and restaurants are equally close. The ocean side is a quick drive (or longer walk) over the ridge. The house was very well set up for small children, with a great outdoor BBQing and dining area and even a trampoline.
For anyone super-sensitive to privacy and/or noise, do bear in mind there's the main house on the same plot (the owners were really helpful and not intrusive at all!), and it's right next to the main road (which doesn't lead anywhere other than Palm Beach, so don't expect any heavy trucks driving past often, just beachgoers and the occasional Bus).