If you don't whisk the fudge it will be too crumbly and may not set correctly.