That’s up to you.  Any diseased or pest-infested plant material should be removed and destroyed, but the remaining plant debris can be left in place for overwintering pollinators or removed for a tidier appearance.