Some code you hadn't given much attention to will take a lot of time. But you can keep that code integrity elsewhere when it doesn't slow you down. If you start out writing assembly code, then you'll end up doing it everywhere, including the ninety-five percent where you don't need it. Remember programs age and become obsolete quickly because CPU's double in speed every year or two. If you aren't planning to take advantage of the next speedup by improving your designs, then your code is going to be obsolete much sooner.
Premature optimization is the root of all evil.
I'm not against optimization. I'm just against unnecessarily poor designs that are justified in the name of optimization. Correct code will not surprise you with so many latent bugs long after you thought you were finished. Finally, I think premature optimization is counterproductive and in the long run actually hinders optimization. Premature optimization will distort designs, destroy encapsulation and information-hiding, and will make code much harder to modify.