The history of the town dates back to the early Middle Ages, when it was a Western Slavic gord. Since the 12th century, until 1789, Iłża belonged to the Catholic Bishops of Kraków. The settlement was twice destroyed by the Mongols (1241, 1260) during the first and second Mongol invasion of Poland. In around 1294 it received Magdeburg rights town charter. In 1340, a stone castle was built here by Bishop Jan Grot, which was expanded in the 15th and 16th centuries. In the 16th century, Iłża became famous for its potters and other artisans or craftsmen. The town prospered, together with whole Polish–Lithuanian Commonwealth. It was conveniently located on a merchant route from the heartland of Poland to the Vistula ports at Solec nad Wisłą, Zawichost, and Sandomierz. In 1576, a town hall was built at the main market square, Iłża had a defensive wall, and several Polish kings visited the castle. The decline of Iłża was brought by the Deluge (1655–1660), when Swedish and Transylvanian armies completely destroyed the town and the castle.