Yes, you can leave the garden grounds and return whenever you want. However, if it is one of those special days when visitors have to buy tickets to enter, you must keep your ticket with you for re-entry.