Not at all! This is a fun little step to make the custard match the jelly, lamingtons and strawberries. But you can for sure leave it yellow if you want to make life easier. Instead of removing the 1/2 cup of jelly mixture, just leave it in the trifle bowl to set with the rest of the jelly.