In the Third Partition of Poland, in 1795, the village was annexed by Austria. Following the Austro-Polish War of 1809, the village was regained by Poles and included within the short-lived Duchy of Warsaw. After the duchy's dissolution, in 1815, the village fell to the Russian Partition of Poland. In 1827, it had a population of 204, which grew to 293 until the late 19th century. On April 19, 1863, Grzybowa Góra was the site of a battle of the Polish January Uprising. A Polish insurgent unit led by Ludwik Michalski defeated a three times larger Russian unit, and the surviving Russian soldiers retreated in panic to Radom. Following World War I, in 1918, Poland regained independence and control of the village.