Ingelfinger was born in Dresden, Germany, the only child of Eleanor Holden and Joseph Franz Ingelfinger. He came to the United States with his family in the early 1920s to live in his mother's home town of Swampscott, Massachusetts where his German father established a general practice as a physician. After initially wanting to enter the business world, faced with dwindling job opportunities after the Wall Street crash he decided to follow in his father and go into medicine. Ingelfinger earned diplomas from Phillips Andover Academy, followed by Yale University in 1932 and Harvard Medical School in 1936.