Billings visited the White House for most weekends during the Kennedy Administration. When a butler commented on the fact that Billings was leaving his belongings in one of the third-floor guest rooms, the First Lady replied: "He's been my house guest since I was married." Sometimes he stayed for longer periods. When the First Lady was away, Billings organized White House dinner parties for the President and old friends, and when the President traveled he kept the First Lady company. One presidential aide later said that "some people saw him so much they thought he was the Secret Service." Billings never had a White House pass and said: "Jack and Jackie were so nice about this that I didn't even have to tell them whether I was coming or going." Historian Sally Bedell Smith compared him to Leonard Zelig, a nondescript character in Woody Allen's 1983 film who is always present in the back row at major events. He sat with the President's family at the Kennedy inauguration and walked not far behind his widow at the Kennedy funeral.