Born in Alameda, California in 1893, he was orphaned at the age of five. After a year in an orphanage, he went to live with his Aunt Lily Von Schmidt, an artist in her own right, and her second husband, Major Charles Lee Tilden, who had been a "forty-niner", and founder of Tilden Park in Alameda County. As a youth, von Schmidt worked as a cowhand and a construction worker. In 1920 and 1924, he was on the United States Olympic Rugby team along with his cousin Charles Lee Tilden Jr. Although the United States team won the gold medal both years, von Schmidt did not play in the only game in 1920, and was sidelined by an injury in the final practice in 1924.