If you can't stand the idea of raisins in your fudge, go ahead and leave them out. I love the contrasting texture and flavor!