The nickname "Little Jemmy" was a reference to Madison's small stature, as he stood at only 5 feet, 4 inches tall.