Harry Clavering is the only son of Reverend Henry Clavering, a well-to-do clergyman and the paternal uncle of the affluent baronet Sir Hugh Clavering. At the novel's beginning, Harry is jilted by his fiancĂŠe, the sister of Sir Hugh's wife, who proceeds to marry Lord Ongar, a wealthy but debauched earl.
Harry's father urges him to make the church his profession; but Harry aspires to become a civil engineer, of the type of Robert Stephenson, Joseph Locke, and Thomas Brassey. To this end, he becomes a pupil at the firm of Beilby and Burton.
A year and a half later, Harry has become engaged to Florence Burton, the daughter of one of his employers. He presses her for an early marriage; but although she loves him deeply, she refuses, insisting that they wait until he has an income adequate to support himself and a family.
At this point, Lord Ongar dies, and his widow returns to England. Sir Hugh, her nearest male relative, is a hard and selfish man, and refuses to see her upon her arrival. This lends spurious credence to rumours about her conduct; and it forces her sister, Lady Clavering, to ask Harry to assist her when she returns.
Harry fails to tell Lady Ongar of his engagement; and, in a moment of weakness, he embraces and kisses her. This puts him in a position where he must behave dishonourably toward one of the two women in his life: either he must break his engagement, or he must acknowledge that he has gravely insulted Lady Ongar. Although he loves Florence Burton and knows that she is the better woman, he is unwilling to subject Lady Ongar to further misery.
Lady Ongar, because of her considerable wealth, is pursued by others. She is courted by Count Pateroff, one of her late husband's friends, and by Archie Clavering, Sir Hugh's younger brother. Count Pateroff's scheming sister Sophie Gourdeloup, the only woman who will see Lady Ongar because of the rumours about her conduct, wants her to remain single so that Mme. Gourdeloup can continue to exploit her.
Mme. Gourdeloup sees to it that Lady Ongar learns about Harry's engagement. Meanwhile, Florence Burton learns that Harry has been seeing Lady Ongar regularly, and decides that she must release him if he does not truly love her.
Through the good influence of his mother, Harry comes to realise that Florence Burton is the better woman and the less deserving of dishonorable treatment. To her letter offering to end their engagement, he responds with a reaffirmation of his love for her. He also writes to Lady Ongar, regretting his past conduct toward her and making it clear that he intends to remain true to his fiancĂŠe.
Soon thereafter, Sir Hugh and Archie Clavering are both drowned when their yacht goes down off Heligoland. This makes Harry's father the new baronet and the possessor of Clavering Park, with Harry the heir apparent. This increase in wealth allows him to marry immediately and to give up engineering, a profession for which he almost certainly lacked sufficient self-discipline. Lady Ongar gives up much of her property to the family of the new earl, and retires into seclusion with her widowed sister.

Question: Why Mme Gourdeloup the only woman who will see Lady Onger?
Answer: Due to the rumors about Lady Onger.
Alternative answer: Because of the rumors about her conduct.