Trifle tends to gets soggy for a couple reasons. First watch out for the orange juice and custard that we are adding both of which contain moisture. You have to think of the amount of these in totality when you are layering a trifle. Secondly, the time of chilling. The longer you chill, the softer the brownies start getting, dont chili the trifle beyond 2-3 hours to prevent sogginess.