Yes, you should be worried about a potential lawsuit if you did not obtain a liability release from the other driver.  If the other driver later developed whiplash symptoms and had to go to a chiropractor he is most likely going to seek out a personal injury attorney to make a demand on your insurance company, which you apparently never even told about the accident.  If he doesn't know your insurance information, the attorney may just go ahead and sue you, figuring you will turn the suit over to your insurance company.  You should alert your insurance company now of what happened, make a record with them, and then hope you do not get sued.