For fudge, it's more about the texture. If the fudge gets hard or dries out, it needs to be thrown away. Also, if it starts to melt or seems slimy, toss it.