Small optimization for the pattern optimizer