Von Fersen returned to Sweden, from where he was forced to watch the ever-increasing expansion of the French revolutionary empire, and in late December 1793, he was suspected of possible complicity in Baron Armfeldt's conspiracy to deprive the Duke of Södermanland of the regency. As a result, von Fersen was deprived of his diplomatic appointments and his post as ambassador. In November 1796, Prince Gustav was declared of age and become King Gustav IV Adolf. His accession restored von Fersen, and other supporters of Gustav III's, back to favour at court. Von Fersen and his best friend, Baron Taube, became two of the most influential advisers to the young king, inculcating in him a "steadfast opposition to Revolutionary France, close relations with Russia, and hostility to Denmark, with the ultimate objective of acquiring Norway".