Ryumin was born in a peasant village in the Kuban area. Reputedly, his family were wealthy farmers, his brother and sister were convicted thieves, and his father in law fought against the Bolsheviks in the White Army commanded by Admiral Kolchak. Despite his background, he survived the wholesale arrests of 'kulaks' instigated by Josif Stalin, and by about 1931 was working as bookkeeper on a collective farm in the Urals. His next break came after Nikolai Yezhov ordered the mass arrest of NKVD officers suspected of loyalty to his predecessor, Genrikh Yagoda in 1937, when Ryumin joined the NKVD as a bookkeeper. He worked for SMERSH during the war, and in 1948 was transferred to the Department for Specially Important Cases, within the MGB.