Kaulbars was born in Mödders in the Governorate of Estonia to Baron Hermann Wilhelm von Kaulbars and Alexandrine Emilie and had an older brother, Nikolai Reinhold Friedrich. Alexander and Nikolai grew up in Saint Petersburg and was of Lutheran faith. He came from the Baltic German noble of Pomeranian origin. The family was of Swedish nobility. Both his father and his brother rose to the rank of general in the Imperial Russian Army. He was educated at the Nikolaev Cavalry School. His first experience in combat was in 1861, while serving with the Egersky Guards Regiment in the suppression of the Polish Uprising.