hbutani/SQLWindowing — Issue #4: query parsing error not detected till the end
Author: achaubal
Created: 2012-02-02 01:43:59+00:00
State: open

Hi,

While running a query, I found that even after completing the CTAS step, the windowing process plods along without any complaint, only to fumble at the end in the reduce stage if a column name is misspelt or does not exist. Shouldn't the parsing in detect that the final select, after the "with" clause has columns that are erroneous? Cause the user has no way of knowing that something errored out unless he looks at the logs. There is a mysterious spew on the screen something like this:
attempt_201201151510_0169_r_000001_0: Query:
attempt_201201151510_0169_r_000001_0: tableInput=(hiveTable=WindowingTempTable_1328129373936)
attempt_201201151510_0169_r_000001_0: partitionColumns=bc_prod_acct_no
attempt_201201151510_0169_r_000001_0: orderColumns=bc_prod_acct_no ASC, bc_date ASC
attempt_201201151510_0169_r_000001_0: funcSpecs=[rank(alias=r, param=[], type=null, window=null),
attempt_201201151510_0169_r_000001_0: avg(alias=ma4_ratio, param=[expr=cv_tot_bal/cv_rtl_bal_am], type=null, window=window(start=range(3 PRECEDING), end=currentRow)),
attempt_201201151510_0169_r_000001_0: last_value(alias=prev_ratio, param=[expr=cv_tot_bal/cv_rtl_bal_am], type=null, window=window(start=range(1 PRECEDING), end=currentRow)),
attempt_201201151510_0169_r_000001_0: ntile(alias=percentile_5, param=[intValue=5], type=null, window=null)]
attempt_201201151510_0169_r_000001_0: select=bc_prod_acct_no, bc_date, cv_ratio, r, ma4_ratio, prev_ratio, percentile_5
attempt_201201151510_0169_r_000001_0: whereExpr=null
attempt_201201151510_0169_r_000001_0: tableOutput=(output(path=ma_6_2011_08, serde=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, serDeProps={field.delim=,, columns=bc_prod_acct_no,bc_date,cv_ratio,r,ma4_ratio,prev_ratio,percentile_5, columns.types=bigint,string,double,int,double,double,int}, format=org.apache.hadoop.mapred.TextOutputFormat)

thanks

ameet