After the death of his brother his parents separated and a nanny took care of Henri.
Njegova plemiška družina je bila stara in cenjena (bili so potomci grofov Toulouse Lautrec in vikontov Montfa iz departmaja Tarn v južni Franciji, ki so se pogosto poročali med seboj).