Is this correct if the operator is created with BuildLeft instead of BuildRight?  I think that would turn it into a RightSemiJoin.  Perhaps we should just remove the option to build on the other side.  I think you can then also safely simplify this to use a HashSet instead of a HashMap, which will reduce memory consumption significantly.