Just that it becomes clutter when the todos don't get done. When I see todos in a codebase that are many years old, it strikes me as a failed convention.

For a large refactoring like this I leave todos as well, but when I'm done with it (if I remember) I grep back through and either carry out the idea or scrap it. If it's not worth doing now, will it be later?