Smith's early victories in New York and California were seen as an inspiration to future woman riders. In 1973, leading up to the "Battle of the Sexes" tennis match, Smith challenged Bobby Riggs to a horse race, while joking that he would turn down the challenge because "He doesn't want to get himself killed." While many Thoroughbred owners were reluctant to lend their horses to a female rider, Smith found an ally in Alfred Gwynne Vanderbilt Jr., the chairman of the New York Racing Association during her jockey career. Their business connection led to rumors of a romantic involvement between Vanderbilt and Smith.