You may not have sifted your dry ingredients enough. Adding your confectioners sugar and almond flour to a food processor will ensure a fine consistency every time. Also, make sure you are folding in your batter slowly.