Once you layer the different ingredients and finish making your trifle, you need to give it some time. The flavour needs to get that wild mix. You can serve this anytime, but not before at least 6 hours of arranging the whole thing. It is better if your trifle has a day before you serve it.