It was a prosperous family, so much so that Hirst's father retired when only thirty-one years old since at that age he inherited his own father's money. Thomas's mother also came from a family involved in the wool trade and Thomas was the youngest of his parents four sons.
Ito ay isang mapalad ng pamilya, kaya marami kaya na Hirst ng ama kapag nagretiro lamang ng tatlumpung-isa na taon gulang dahil sa edad niya na minana sa kanyang sariling ama pera. Thomas ng ina din nagmula sa isang pamilya na kasangkot sa lana kalakalan at Thomas ay ang bunso ng kanyang mga magulang ng apat na anak.