Poh was born in Singapore in 1930. He is the maternal grandson of prominent businessman Tan Kah Kee. His family fled Singapore before the Japanese invasion and landed in Mumbai, he studied at a Catholic mission school for four years. Poh and his family returned to Singapore after the Japanese surrender in 1945, continued his studies at Raffles Institution and entered the University of Malaya in 1950. He was a founding member of the Socialist Club in 1953 and became its second President from August that year till the following year. Poh was one of the eight members of the Fajar editorial board charged with sedition in 1954. He graduated three years later with a degree in Medicine.