Hillmer was born and raised in Texas. As a 15-year-old boy, he travelled to Chicago with his mother, a cousin and a friend to visit the 1933 World's Fair. It was there, in the California exhibit, that he encountered the redwood boards that would become a key feature of his later architectural practice: four-foot by twenty-foot planks displayed under spotlights to highlight one of that state's key products. Several years later Hillmer received a scholarship to the University of Texas, where he enrolled in architecture. After college, Hillmer joined the air force, and was stationed in San Diego. Hillmer used his time-off during the war to explore the new modernist buildings that were being built in southern California, including renting a room in the Pueblo Rivera Apartments by R. M. Schindler, and living for a time at the La Jolla Art Center by Irving Gill. Toward the end of the war, Hillmer visited Frank Lloyd Wright at Taliesen West in Scottsdale, Arizona. Hillmer was invited by Wright to join his Taliesen Fellowship, but Hillmer declined despite his appreciation for Wright's work.