Yes, you should remove all brown leaves. You can also cut off the brown tips from the leaves to give your plant a fresher look.