You cannot gain entry to the gardens without first paying for entrance to the building itself. Short of a fire or other major emergency the gates to the gardens are always padlocked. The gardens are part of Casa Loma and they are not going to permit free entry into the building just so people can walk into the gardens.