You can use all brown sugar, but it will alter the texture and structure of the frosting. I added powdered sugar to help the frosting keep its shape. Without it, the frosting may not be as stiff or structured.