Brothers of the Sacred Heart
Dooley grew up in Mobile, Alabama and attended the McGill Institute, administered by the Brothers of the Sacred Heart. Dooley competed on behalf of McGill's athletic teams, known as the Yellow Jackets, and for a few years considered basketball to be his best sport.