Your brioche dough should double in size during the slow proof in the refrigerator. Sometimes this doesn't happen. This is almost certainly due to the yeast and may have resulted due to the yeast being under-active. This can happen when the yeast has expired, if the milk was too warm/cold or the dough not kneaded long enough. If your dough hasn't doubled at this stage, continue with the recipe and don't abandon hope. These may still result in risen buttery brioche.