Unless you do not want to hamper the growth of your rosemary plant, trim the flowers as soon as it starts budding. If you want to use rosemary flowers for culinary use, you can let the plant bloom flowers as much as it can.