The size of the lemons is a better indicator of when they need to be harvested than the color. Start picking your lemons once they’ve reached 2-4 inches in size and firm. The lemons will continue to ripen even after they’ve been picked. If you leave them on the tree for too long, the lemons will start to rot and fall off.