There could be a few reasons for this. First off, you might be using brioche that’s a little too fresh. Use stale brioche that’s at least a day old – it’ll hold together better. Also, if your pan is too hot, the outside of the bread might cook too quickly while the inside stays soggy. Try lowering the heat a touch and see if that helps.