The best time to harvest black pepper is in the spring or summer, when you see clusters of peppers forming. You can then collect the peppercorns and dry them for future use.