Yes, you can, and later on, it's essential for fine-tuning your build. Don't worry; you can change your armor appearance when you've advanced through enough quests, so you'll always look cool.