In 1628, Grand Duchy of Lithuania scribe Stefan Krzysztof Pac of the Pac family purchased the building. Their son, Krzysztof Zygmunt Pac, who later became Chancellor of the GDL, became the next owner of the building. In 1655, during the Russo-Polish War, the Palace at St. John Street burnt down. The Palace was rebuilt after the War. After the death of Krzysztof Zygmunt Pac, the palace passed into the possession of his distant relative Michał Kazimierz Pac. The next owner was a cousin of Jan Kazimierz – Józef Franciszek Pac. In 1748, the Baroque Palace burnt down again. Neither Józef Franciszek Pac nor his son Michał Jan Pac, who was the last descendant of the previously rich and influential family, were able to rebuild it. In 1771, for political reasons, Jan Pac was forced to emigrate, and his possessions were sequestered. In 1783, the ruined palace was acquired by the Grand Chancellor of Lithuania, Aleksander Michał Sapieha. He rebuilt the Palace and decorated it in the contemporary Classical style.