Not if you don’t want to. All the ingredients contribute to Dirty Rice's flavor, but none take over. Chicken livers add a subtle undertone to the flavor profile. Many recipes call for using the gizzards along with the livers, so that is certainly an option. Don't go overboard on the amount of liver you use because they can certainly take over the flavor profile if you let them. Most people won’t even know they're there. If you just can't do it, leave them out and you have a wonderful recipe for Rice Dressing!