This happens when there is not enough binder in the mix. I like to use flour, eggs, and cheese as binders. You could also add breadcrumbs. Make sure you don't try flipping the fritters before they are golden and crisp.