Absolutely, do not let perk points burn a hole in your pocket. Many great perks come relatively early, but you're not capable of fully utilizing them until later in the game. You can get by just fine with a skeleton crew build for a while. Once the difficulty sees a significant spike, that's the time to start to deck out your perk build.