Baker is the nephew of Wes Chandler, who is a former All-American Gator wide receiver, NFL standout and professional football coach. Baker's younger brother Perry Baker is a professional rugby sevens player and represented the USA in the 2016 Olympics.