Peter Baker was born in Lewisham, South London; he was nicknamed "Ginger" for his shock of flaming red hair. His mother, Ruby May (née Bayldon), worked in a tobacco shop. His father, Frederick Louvain Formidable Baker, was a bricklayer employed by his own father, who owned a building business, and a lance corporal in the Royal Corps of Signals in World War II; he died in the 1943 Dodecanese campaign. Baker attended Pope Street School, where he enjoyed being on the football team and was considered "one of the better players" and then, after he passed the Eleven-plus, at Shooter's Hill Grammar School. While at school he joined Squadron 56 of the Air Training Corps, based at Woolwich and stayed with them for two or three years.