The different textures and shapes that inanimate objects have can make your garden more interesting. They also provide hiding places for some helpful critters - just like the weedy patch does. If placed out in the open, a rock or pole can provide a nice place for a predator insect or bird to keep watch for prey. I try to leave my flowers that have gone to seed. The seeds can provide much needed food for birds and other creatures during the winter.
Insects lay eggs in and on flower stalks.
If you take them down, some insects will have no place to lay eggs in the fall. Finally, just because your garden looks sloppy doesn't mean it has to be unhealthy. Organic gardeners should take care to clean up the garden to remove old plants that might harbor disease and insects. Well, it's not us small time gardeners who use most of those chemicals. Farmers use a lot more than we do at our home.