James Madison enjoyed ice cream, but one of the desserts that he is linked to is Dolly Madison's Soft Gingerbread. James Monroe – Cry Babies, which are molasses cookies. John Quincy Adams did not have much of a sweet tooth, preferring fruit.