First, you really need five hats and it could take you a couple of years to get to the proper number. Here's why. The Felt. The felt hat starts out as your winter hat and also your formal dress-up hat in summer or winter.