This is usually a sign that the dough has been over-proofed. Over-proofed dough resembles a balloon and the brioche will be very crumbly after baking. Make sure to process the dough when it has doubled in size.