No, the sugar is essential to activating the yeast and giving the brioche a golden color as it bakes. Only a small amount of sugar is added, so it gives the bread only a very slight sweetness. You can omit the vanilla, if desired.