You can leave them out if you like. The casserole won't be the same but it will still be delicious.