Walker was born in Fairbanks, Alaska and raised in the small, rural interior city of Delta Junction and the port of Valdez on Prince William Sound. He was the fourth child of Alaskan pioneers Frances (Park) and Ed Walker. During World War II, Ed was an Alaskan Scout with Castner's Cutthroats in the Aleutian Islands and Frances worked on the Alaska-Canadian Highway. During the 1964 Alaska earthquake, which severely damaged Valdez, the family lost most of their personal and business possessions. At the age of 12, Walker became a janitor to help his family.