Lê was born in 1968 in Hà Tiên, a Vietnamese town near the Cambodia border. The Cambodian-Vietnamese War of the 1970s brought Khmer Rouge troops to the region. When he was ten, in 1978, his family made their escape. When the journey began, Lê was accompanied by his six siblings and his mother. However, while the ten year-old Lê made it safely onto the boat along with his mother, others of his siblings were not so lucky. In an interview published in a catalogue for one of his exhibitions, he recalled the story of the escape. It was a "desperate run for freedom," and he said that he "[would] never forget the look on his mother's face as she scoured the beach...for signs of her eldest sons and daughter." After a brief stay in Thailand, the family eventually moved across the ocean and settled in Los Angeles.