Your fritters may fall apart if there is either too much moisture or they are too dry. If the final batter is too wet add more flour or breadcrumbs, a little at a time, to achieve the proper consistency. If the batter is too dry add milk or water a small amount at a time.