Both of Ayayi's parents are from Benin. His father stands and played basketball professionally in France and for the Beninese national team. Ayayi's older sister Valériane has played for the French women's national team, most notably at the 2016 Summer Olympics, and in the Women's National Basketball Association (WNBA). His younger brother Gérald plays professional basketball in France.