This may have happened because you over stirred and the sugar became grainy. To prevent this, do not stir the fudge as it's cooling. As soon as it starts to boil and bubble, leave it be.