If your banana peppers lack the sweet or spicy flavor you expect, you probably picked them too early. For the peppers to fully develop their flavors, they need to be kept on the plant until they begin to turn yellow.