Unfortunately, because most mashed potato recipes call for copious amounts of butter, cream, and milk, using leftovers is problematic and will more than likely make your fritters soggy and more prone to falling apart.