Agha Ahmad Ali's grandfather Agha Abdul Ali was a Sunni Muslim calligraphist who originated from Isfahan in Iran and settled in the city of Dhaka during Nader Shah's invasion of India. Ahmad's father was Agha Shajaat Ali, a Dhakaiya who had a hobby of collecting rare manuscripts. Ahmad studied Persian locally with Munshi Mutasim Billah as well as Khwaja Asadullah Kawkab, a noteworthy poet of the Dhaka Nawab family. He developed a personal library of over 2000 books. It is said that he completed all "valuable" books in the city of Dhaka some time between 1856 and 1860.