The hamlet of Gradišče on the southwest slope of the hill has the remnants of prehistoric structures, attesting to early settlement of Šmarjetna Gora. During the Middle Ages, Šmarjetna Gora was a property of the Bishopric of Freising, which planted vineyards here. The vineyards were maintained until the 16th century, when they were converted into fields. In the 12th century, the Counts of Ortenburg built Wartenburg Castle on a cliff below the hill. This represented a threat to the Dominion of Škofja Loka, and so Bishop Oton purchased it in 1282 and had it razed. The Ortenburgs later restored the castle and carried out raids against Okroglo and Stražišče, which belonged to the Bishopric of Freising. Wartenburg Castle is now a ruin. Near the castle there was an auxiliary building resembling a manor; it had a turret and bore the year 1653, but was destroyed in the 1895 earthquake. Saint Peter’s Manor ()—also known as Schrottenturn Manor (, ) and Detela Manor ()—was built at the foot of the hill in 1537 by the bishops of Freising, and the door casing bears the year 1574. The adjacent Saint Peter’s Chapel was built around 1500. During the Reformation, Lutherans gathered at the manor, and it contains the graves of the Protestant Siegersdorf family. After the Counter-Reformation, the chapel was remodeled and re-consecrated in 1645. It has a flat coffered ceiling and murals in the apse, a 1615 painting of the Holy Sepulchre by Matija Plainer, and a panel on the north wall describing the famine of 1817.