We can remove Project Rels only when 1) all the expressions in Project Rels
are "transferable" expressions, and 2) in the same order as the input
fields, 3) all the fields will appear in the Project Rels.

For instance:

Project Rel ( Col1, Col2, Col3, Col4, Col5)

Input Rel ( Col1, Col2, Col3, Col4, Col5).

If either of the above 3 conditions is not met, we could not remove the
Project Rel.  If all the conditions are met, since the project expressions
are "transferable",  all the Project OP has to do is to transfer the direct
reference of value vectors, which probably incurs very small cost overhead,
comparing to other operators.

I could put a simple rule to remove Project Rel when the 3 conditions are
good. But I feel it may not impact the overall performance a lot if we
defer it for later.



On Tue, Apr 15, 2014 at 11:21 AM, amansinha100 <notifications@github.com>wrote:

> In
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java:
>
> > @@ -54,7 +57,24 @@ protected Object clone() throws CloneNotSupportedException {
> >    @Override
> >    public PhysicalOperator getPhysicalOperator(PhysicalPlanCreator creator) throws IOException {
> >      StoragePlugin plugin = this.drillTable.getPlugin();
> > -    GroupScan scan = plugin.getPhysicalScan(new JSONOptions(drillTable.getSelection()));
> > +
> > +    boolean containStar = false;
> > +    final List<String> fields = getRowType().getFieldNames();
> > +    List<SchemaPath> columns = Lists.newArrayList();
> > +    for (String field : fields) {
>
> Yeah, I meant the same column appears multiple times in different
> clauses...so if Optiq is taking care of it then I suppose you don't need to
> check for the duplicates or the constant exprs. The only other comment I
> had was this is actually column push-down, not project push-down since you
> are not getting rid of the Project Rels from the plan. In theory, after the
> columns have been pushed into the scan, if the Project above the Scan is
> producing the exact same set of columns as the scan, then there is no
> reason to keep the Project. Do we want to address this now or defer it for
> later ?
> … <#145669f5ecdd4bee_>
> On Tue, Apr 15, 2014 at 10:55 AM, jinfengni <notifications@github.com>wrote:
> > In >
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java:
> > > > @@ -54,7 +57,24 @@ protected Object clone() throws
> CloneNotSupportedException { > > @Override <https://github.com/Override>> > public PhysicalOperator getPhysicalOperator(PhysicalPlanCreator
> creator) throws IOException { > > StoragePlugin plugin =
> this.drillTable.getPlugin(); > > - GroupScan scan =
> plugin.getPhysicalScan(new JSONOptions(drillTable.getSelection())); > > + >
> > + boolean containStar = false; > > + final List<String> fields =
> getRowType().getFieldNames(); > > + List<SchemaPath> columns =
> Lists.newArrayList(); > > + for (String field : fields) { > > By "duplicate
> columns", do you mean the same query appears multiple time > in Select
> clause, or order by clause, group by, etc...? If that's the case, > optiq
> will take care of the duplicate columns in ProjectRel, SortRel, etc, >
> since optiq uses index based RexNode to represent the expression. The Scan
> > RelNode's RowType will have only the identical columns. For example, for
> > this query: select R_REGIONKEY, R_NAME, R_NAME, R_REGIONKEY, R_REGIONKEY
> + > 100 FROM REGION... Scan will have RowType (R_REGIONKEY, R_NAME), while
> > ProjectRel will have different expressions referring to the same index in
> > the input's RowType. DrillProjectRel(R_REGIONKEY=[$0], R_NAME=[$1], >
> R_NAME0=[$1], R_REGIONKEY0=[$0], EXPR$4=[+($0, 100)]) For constant >
> expressions, ProjectRel will take care of the constant expression. ScanPrel
> > will only have to deal with column fields. > ... <#1456687fb3827ffb_> >
> On Tue, Apr 15, 2014 at 9:07 AM, amansinha100 <notifications@github.com>wrote:
> In
> exec/java-exec/src/main/java/org/apache/drill/exec/planner/physical/ScanPrel.java:
> > @@ -54,7 +57,24 @@ protected Object clone() throws
> CloneNotSupportedException { > @Override <https://github.com/Override> <
> https://github.com/Override> > public PhysicalOperator
> getPhysicalOperator(PhysicalPlanCreator creator) throws IOException { >
> StoragePlugin plugin = this.drillTable.getPlugin(); > - GroupScan scan =
> plugin.getPhysicalScan(new JSONOptions(drillTable.getSelection())); > + > +
> boolean containStar = false; > + final List<String> fields =
> getRowType().getFieldNames(); > + List<SchemaPath> columns =
> Lists.newArrayList(); > + for (String field : fields) { At the time of
> adding the columns, I think we should check for duplicates since we don't
> want duplicate columns to be produced by the underlying scan. Also, I am
> wondering if we should check for constant expressions...or is that already
> taken care of before getting here. -- Reply to this email directly or view
> it on GitHub<
> https://github.com/mapr/incubator-drill/pull/1/files#r11642143> . --
> Reply to this email directly or view it on GitHub<
> https://github.com/mapr/incubator-drill/pull/1/files#r11647820> .
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/mapr/incubator-drill/pull/1/files#r11649376>
> .
>