Your mixture is either too dry or too cold for binding. To prevent that, bind the ladoos just as the mixture is comfortable to handle. To fix the dryness, add a tablespoon of milk or ghee as needed to bind.