Usually that will be the lower leaves. Just cut away any unsightly foliage. Daylilies don't need to be deadheaded to encourage more bloom. But the bloom from the previous day may not fall away neatly. A pass through your garden, picking off any spent bloom will make your garden look well manicured.
Some daylily varieties seem to discard yesterdays bloom better than others.
When removing the spent bloom, be careful not to snap off a young blossom. To encourage bloom and to prevent unnecessary stress on your daylily plant, remove any seed pods that may develop. Click here to see a seed pod Cut off spent flower scapes when all the bloom are finished on that scape. Click here to see a scape illustration. You can cut back a spent scape while other scapes are still blooming.