I suggest removing the talk header if there were actually zero talk page comments (other than your own).  I'd check the article to see what ref style it's using, and leave the line about refs if the article is using footnotes (which I suspect will be nearly all of them that contain any refs), and remove it only if a different style is being used.  
IMO spamming the talkheader template is in the nature of a venial sin:  In the future, you should use the template when appropriate; as for past mistakes, it might be worth cleaning them up  but there might well be more important things to do, in which case you should focus on the more important work.  Unnecessary talk headers can be removed whenever you're otherwise editing a talk page for a 'real' purpose, after all.