Goff attended a dame school in Basingstoke until he was eight. Thereafter, he attended St Aubyns School, Rottingdean, and started at Eton College in September 1939, at the beginning of the Second World War. While at Eton, he focused on classical languages and history, preferring these to science subjects. There, Henry Ley, an organist who had played at the coronation of King George VI, encouraged his love for the piano and taught him to play. He left Eton in December 1944, having received a deferred offer of admission to New College, Oxford, for after he completed his military service.