Or, worse, reprofiling and recompilation. In terms of control theory, since profiling information is usually not gathered on the actual program, it can be considered an open loop control system with closed loop feedback control involving a model - a not necessarily very accurate model. Control theorists will be well aware of the bad implications of this for stability and performance. Software optimization research should therefore be targetted at improving the feedback loop. The big advantage that software has over hardware from the point of view of dynamic optimization is persistent storage of the optimized code.
Software optimized code can survive from one user to another.
Hardware optimized code can pretty much only survive as long as the code is resident in the instruction cache, or other on-chip data structure. The question is not whether optimizations should be done in hardware versus software. Hardware and software mechanisms seem to have intrinsic advantages and disadvantages, e.g. with respect to speed. I look forward to software optimizers exploring more dynamic optimization and fast feedback techniques.