Asad was a member of the Qasr clan, a subtribe of the Bajila. His great-grandfather, Asad ibn Kurz al-Qasri, is said by some traditions to have been the chief of the Bajila in the times of the Islamic prophet Muhammad, and is counted as one of Muhammad's companions. Other traditions, hostile to the family, report that Asad was a Jew and a runaway slave. Asad's grandfather Yazid was an early and prominent supporter of the Umayyads in the First Fitna, while Asad's father Abdallah sided with Ibn al-Zubayr in the Second Fitna, but was eventually pardoned by the Caliph Abd al-Malik ().