You should remove them. "Deadheading" spent blossoms and seedheads will encourage fresh growth. Trim them off at the point where they meet the main stem, or at the nearest leaf node.