No, you can build a village whenever you like, but during the Village Master and Village Mania events, the cost is the lowest.