He was the son of Victurnien-Jean-Baptiste de Rochechouart, duc de Mortemart (1752–1812), and his second wife, and Adélaïde de Cossé-Brissac (1765–1820), only daughter of Louis-Hercule-Timoléon de Cossé, duc de Brissac (1734–92), and Adélaïde-Diane-Hortense-Délie Mancini (herself daughter of the duc de Nivernais). He and his family left France in 1791 and Casimir grew up in England, returning to France with his mother in 1801. Despite belonging to one of the old noble families of France, he backed the First French Empire, joining the "gendarmes d'ordonnance" (commanded by the count of Ségur) in September 1803. He moved to the 1st Dragoon Regiment on 10 February 1806 and took part in the campaigns in Prussia (1806), Poland (1807), Austria (1809) and Russia (1812), fighting at Golymin (1806), Pułtusk (1806), Heilsberg and Friedland, where he proved his sang-froid and firmness under fire by beating off Russian attacks. He was made a member of the Légion d'Honneur on 1 October 1807, a lieutenant in the 25th Dragoon Regiment and "aide-de-camp" to general Nansouty on 2 and 10 March 1809. He finally became a captain in the 25th Dragoon Regiment on the following 26 July.