Colbert left Junot in 1805 and followed major général Maréchal Berthier as his aide de camp. He assisted at Austerlitz, where he was wounded and made a chef d'escadron. He fought bravely at Iena and Pułtusk. In 1807 he was made a colonel and head of his old regiment, the 7th Hussars. This unit was now part of Lassalle's brigade, nicknamed the "infernal" brigade due to its passion. Napoleon made him a knight of the Légion d'honneur in 1808 and baron de Chabanais et de l'Empire in 1809