No one really knows Felix Bush (Robert Duvall), who lives as a hermit deep in the woods. Rumors surround him, such as how he might have killed in cold blood, and that he's in league with the devil. So the town is surprised when Felix shows up in town with a fat wad of cash, requesting a "funeral party" at Reverend Gus Horton's church for himself. Rev. Horton declines this proposal. Frank Quinn (Bill Murray), the owner of the local funeral parlor in financial trouble, coveting Bush's wad of cash, agrees to advertise a funeral party at which the townsfolk will be invited to tell Felix Bush the stories they've heard about him. To ensure a good turnout, a lottery is organized, with Bush's property as the prize. Many people buy tickets.
Things get more complicated when an old mystery is remembered, involving a local widow named Mattie Darrow (Sissy Spacek), who was Bush's girlfriend in their youth, and her deceased sister, Mary Lee Stroup (Arin Logan). With the help of a preacher who insists on Bush telling Mattie the truth, Bush tells those gathered at his funeral party (including Mattie) what happened forty years earlier. He reveals his affair with Mattie's married sister, Mary Lee. He confesses to Mattie that it was Mary Lee who was his true love, his only love. They made plans to run away together, and when she didn't arrive at the agreed place, he went to her home to search for her. He discovered that her husband had attacked her with a hammer, knocking her out. The husband threw a kerosene lamp against a wall to set the house on fire and kill himself, the unconscious Mary Lee, and Bush. Bush freed himself from the attacking husband, but as his clothes caught fire, he also saw Mary Lee catch fire. As he went to put the fire out, he felt himself flying through the window, possibly pushed by the husband, and he was unable to re-enter the house to save Mary Lee.
Mattie returns, seeming to forgive Bush. He sees what appears to be the ghost of Mary Lee down the lane. He dies shortly after his funeral party. His actual funeral service is held on a burial plot on his property. Charlie officiates the ceremony, with Reverend Gus Horton, Buddy, his wife and child, Mattie and Frank in attendance. After a short benediction from Charlie, Mattie places a portrait of her sister, Mary Lee on Felix's casket, signifying her forgiveness of his past actions and allowing them to be together, even if only figuratively. The mourners leave Felix's property as his grave is filled.

Question: Who did Felix have an affair with?
Answer: Mary Lee
Alternative answer: Mary Lee, Mattie's sister