apache/spark — pull request #2109 by marmbrus (2014-08-24 20:35:17+00:00)
Title: [WIP][SPARK-3194][SQL] Add AttributeSet to fix bugs with invalid comparisons of AttributeReferences

It is common to want to describe sets of attributes that are in various parts of a query plan. However, the semantics of putting `AttributeReference` objects into a standard Scala `Set` result in subtle bugs when references differ cosmetically. For example, with case insensitive resolution it is possible to have two references to the same attribute whose names are not equal.

In this PR I introduce a new abstraction, an `AttributeSet`, which performs all comparisons using the globally unique `ExpressionId` instead of case class equality. (There is already a related class, `AttributeMap`) This new type of set is used to fix a bug in the optimizer where needed attributes were getting projected away underneath join operators.

I also took this opportunity to refactor the expression and query plan base classes. In all but one instance the logic for computing the `references` of an `Expression` were the same. Thus, I moved this logic into the base class.

For query plans the semantics of the `references` method were ill defined (is the the references output, or used by expression evaluation, or what?). As a result, this method wasn't really used very much. So, I removed it.

TODO:
- [ ] Finish scala doc for `AttributeSet`
- [ ] Scan the code for other instances of `Set[Attribute]` and refactor them.
- [ ] Finish removing `references` from `QueryPlan`