A grainy texture can happen to your peanut butter fudge if crystallization occurred during the cooking process. This happens when sugar crystals form on the side of the pot, then fall into your mixture...which is why it's important not to stir while it's boiling.