He was the son of Colonel John Drummond of the Royal Horse Artillery and his wife (or lover) Gertrude Drummond. John died at age 55, only three months after Jack's birth. Jack was adopted and raised by John's sister Maria Spinks, who lived in nearby Charlton. Maria's husband, George, was a retired captain quartermaster, who had seen action in the Crimea. According to James Fergusson, life could not have been much fun for the solitary boy in the elderly couple's home.