The egg and buttermilk help bind the ingredients into a somewhat sticky batter. Make sure that you have enough oil in the frying pan so the fritters don't rest on the bottom. Also, properly preheat the oil and don’t overcrowd the skillet, which causes the oil temperature to drop rapidly when you add the batter. Also, check that the oil returns to the proper temperature between batches.