Laxer was born in Montreal, Quebec, on 22 December 1941 and was the son of Edna May (née Quentin) and Robert Laxer, a psychologist, professor, author, and political activist. His father was Jewish and his mother was from a Protestant family. Both were members of the Communist Party of Canada and its public face, the Labor-Progressive Party, with Robert Laxer being a national organizer for the party. The Laxers left the party, along with many other members, following Khrushchev's Secret Speech revealing Joseph Stalin's crimes, and the 1956 Soviet invasion of Hungary. James Laxer wrote about his experiences growing up during this period in his memoir "Red Diaper Baby: A Boyhood in the Age of McCarthyism". His father came to serve as a significant influence on his political worldview.