Joseph Barnett was born on 25 May 1858 in Whitechapel's Hairbrain neighbourhood. He was the fourth son of dock worker John Barnett and his wife, Catherine, Irish immigrants who fled their native country for Great Britain following the great famines. The Barnetts already had a son before they moved to Whitechapel, Denis, and went on to have three others (excluding Joseph) - Daniel (b. 1851), Catherine (b. 1853) and John (b. 1860).