In 1835 Count Tolstoy received an appointment at the Ministry of Interior and gave his resignation in 1840. In April 1851, he was appointed governor of Kaluga, and on 27 April 1854 - governor-general of the city of Taganrog. Tolstoy held this office until September 1856 and participated in the defense of the city from bombardments and landing operations during the Siege of Taganrog in 1855. On 31 August 1859 he was appointed governor of Penza and on 4 August 1861 – Russian senator. In 1870, Tolstoy was decorated with an Order of St. Alexander Nevsky for the 50 years of service.