First documented mention of Skaryszew comes from the year 1198, when the village belonged to the Abbey of Order of the Holy Sepulchre, located at Miechów. The monks opened here a branch of their abbey, and probably in the late 12th century, a wooden church of St. Jacob was built, together with a house for the monks. Due to the efforts of the Order of the Holy Sepulchre, merchants and skilled artisans were attracted to the quickly growing village. Skaryszew was destroyed in the Mongol invasion of Poland, and soon afterwards, Prince of Kraków and Sandomierz Bolesław V the Chaste granted the village the so-called Środa Śląska town charter ("Neumarkt-Magdeburger Recht"), based on the charter of Nowy Korczyn (see also Magdeburg rights). The charter was confirmed by King Casimir III the Great in 1354.