Change default for USTAT_USE_IS_WHEN_NO_STATS

This cqd was originally set to OFF by default, causing
internal sort not to be used in cases where there are
no existing histograms. Internal sort is known to
underperform when there are a small number of unique
values for a column, and without prior histograms, the
unique entry count is unknown.

However, experience with POCs has shown that using internal
sort in these situations (unique count unknown) is in
general significantly faster than not using it, so the
default setting has been changed to ON.

Running developer regressions with this setting exposed a
bug that occurs when internal sort is used on an empty table
that previously did contain rows. This change includes a known
diffs file for the failing test until the bug (LP 1393930)
can be fixed.

Change-Id: I9ff6672a168016fb83fadb26b40a30c7fa3eee6c
Closes-Bug: #1392555
Native external hbase table access (select, IUD) changes.

-- IUD on external hbase tables is now enabled by default
-- predicates on native hbase tables can now be pushed down to
   hbase region server
-- traf varchar col maxlength is now 200K by default,
   can be changed by cqd max_character_col_size
-- executor handles column values length greater than 32K during
   move to/from JNI
-- error is correctly returned if data retrieved from hbase exceeds expected
   max row length
-- hbase column_create function now takes an expression/param as its
   column name operand

Change-Id: Ieb3fcabfebaa22008eff2a049fc1e2000e68861e
Port change to get stats from memory rather than RMS,
which is expensive.

Picked up a change from Selva to handle the core seen
- missed in previous merge.

Change-Id: I4426b58e1fe18492f52ed577e2c38b1453feff17
Merge "Port change to get stats from memory rather than RMS, which is expensive."
Merge "Change default for USTAT_USE_IS_WHEN_NO_STATS"
LOB datatype infrastructure support

    Technology preview.
    More changes expected as part of this work before it
    is user ready.
    blueprint lob-support

    This checkin contains basic support for create blob/clob datatypes.
    The feature is disabled by default. Instructions on how to enable are
    listed below.
    New test executor/TEST130 that turns the feature on and tests out the
    functionality.
    New mxlobsrvr process will be started as part of sqstart.
    Create and drop of  tables with LOB columns.
    No support for alter.
    DML support for LOB datatypes.
    Insert, update and deletes. Joins of 2 tables with LOB columns are
    allowed but joins on LOB columns temselves are not allowed.
    Insert-select from one LOB table another not yet supported.
    Link to document from the blueprint will be added shortly.

    To enable and try LOBs:
    On a developer workstation :

    cqd TRAF_BLOB_AS_VARCHAR 'OFF';

    On a cluster after installing the code 2 steps are needed:

    1. cqd TRAF_BLOB_AS_VARCHAR 'OFF';
    2. sudo su hdfs --command "hadoop fs -mkdir /lobs"
       sudo su hdfs --command "hadoop fs -chown -R trafodion:trafodion
    /lobs"

        This checkin includes several merges from the mainline and each of
    the lines below represents one commit to the project branch where this
    work was done. `
        -Turning off LOB code by default. But turning it on in
    executor/TEST130 just to ensure testing the code path.
        -Support for showddl and some syntax for external files and stream.
        -LOB regression test
        -Workaround for dtm issue LP 378167
        -Changes to make append work. Changes to use lob heap.
        -Fix for using system heap for all LOB allocations and handling NULLs.
        -Added workaround for cursor delete issue. LP 1376969
        -Fixes for update.
        -Parser changes for exe_util_lob_extract
        -Pull in lob extract code
        -Adding mxlobsrvr directory

        -Fixed the LOB interface to use  2 new params for cursor fetches. They do
        not overload the LOB Handle and LOB handle length anymore.
        Added a flag to lobGlobals to indetify it's a hive access.
        Cleaned up parser code.

        -LOB support for create,drop,insert,delete,select.

Change-Id: I7c8125696e847b71580b746388632e75741bd347
removal of obsolete code that is no longer valid and unused in Trafodion

Change-Id: Ie9212627d102661fe1a7b02fd6e9ae9a23198f2a
fix for hive/test018

Query caching seems to cause hive/test018 to fail
disabling it for now till the query caching
issue is fixed

bug 1396386

Change-Id: Ife834050ce8f7d7b968488f5a5ff3f189ac2b666
Merge "removal of obsolete code that is no longer valid and unused in Trafodion"
Merge "fix for hive/test018"
Bug fixes

-- fix for LP bug 1396311: failed primary key constraint was
   not cleaning up metadata entry
-- removed obsolete code reference in gui debugger
-- disabled display of software version stored in metadata until
   install scripts add enhancement to update metadata with
   current software version

Change-Id: I3922e689feec1d0e048083b9a36e6b43f2def49d
Minor tracing updates, addresses TransactionState leak

Change-Id: I75cb9f2486404b3e6cbce39247bbdf2b668d4562
SQL syntax to cancel executing query, phase 1

This check-in implements the command "control query cancel qid <qid>".
The command will send a cancel message to the MXSSMP cancel broker,
but since the master executor is not yet sending a query started
message,
no query will be canceled, and if the message is successful SQLCODE
-8026 will be raised. The change also validates the cancel command.
produces new EXPLAIN and SHOWPLAN output and handles errors, as shown
in a new regression test.

Change-Id: I472fca043aa12d684c13581264baf22a7ea5f49b
Implements: blueprint sql-query-cancel
Support for divisioning (multi-temperature data)

This is the initial support for divisioning. See
blueprint cmp-divisioning for more information:
https://blueprints.launchpad.net/trafodion/+spec/cmp-divisioning

Also, this change fixes the following LaunchPad bugs:

Bug 1388458 insert using primary key default value into a salted
            table asserts in generator

Bug 1385543 salt clause on a table with large number of primary
            key columns returns error

Bug 1392450 Internal error 2005 when querying a Hive table with
            an unsupported data type

In addition, it changes the following behavior:

- The _SALT_ column now gets added as the last column in the
  CREATE TABLE statement, rather than the first column after
  SYSKEY. The position of _SALT_ in the clustering key does
  not change. This will cause some differences in INVOKE and
  in the column number assigned to columns.

- For CREATE TABLE LIKE, the defaults of the WITH clauses
  are changing. CREATE TABLE LIKE now copies constraints,
  SALT and DIVISION clauses by default. The WITH CONSTRAINTS
  clause is now the default and should no longer be used.
  Instead, WITHOUT CONSTRAINTS, WITHOUT SALT and WITHOUT
  DIVISIONING clauses are supported.

- For CREATE INDEX ... SALT LIKE TABLE, we now give a
  warning instead of an error if the table is not salted.

- Also added an optimization for BETWEEN predicates. If
  part or all of them can be converted to an equals predicate,
  we do this now. Example:
  (a,b,c,d) between (1,2,3,4) and (1,2,5,6)
  is transformed into
  a=1 and b=2 and (c,d) between (3,4) and (5,6).

More detailed description of changes:

- arkcmp/CmoStoredProc.cpp
  sqlcat/desc.h
  + other files

  Using the new FLAGS column in the COLUMNS metadata table to store
  whether a column is a salt or divisioning column. Note that since
  there may be existing salted tables without this flag set, the flag
  is so far only reliable for divisioning columns.

- comexe/ComTdb.h
  comexe/*.h
  generator/Generator.cpp
  sqlcomp/CmpSeabaseDDLmd.h:
  Changed the column class field in struct
  ComTdbVirtTableColumnInfo from a string to the corresponding
  enum. Sorry, this caused lots of small changes (deleting "_LIT"
  from the initializers). Also added the column flags.

- executor/hiveHook.cpp: Added a check for partitioned tables
  (having multiple SDs). This is part of the fix for
  bug 1353632.

- GenRelUpdate.cpp: When generating the key encoding expression
  for an insert inside a MERGE operation, we assumed the new
  record expression was in the order of the key columns. Added
  a step to sort by key column, so we can pass the expression
  in any order.

- optimizer/ItemExpr.cpp
  optimizer/ItemNAType.h:
  Added a named NATypeToItem item expression.
  This is used to do a primitive "bind" operation of an item expression
  when processing a DDL statement. Specifically, to bind the DIVISION BY
  clause in a CREATE TABLE statement.

- optimizer/ItemFunc.h
  optimizer/SynthType.cpp: The DDL time "binder" gets expressions as
  they come out of the parser, e.g. a ZZZBinderFunction. Need to add
  type synthesis for some cases of the ZZZBinderFunction.

- optimizer/NATable.cpp
  Removing some dead code. Adding an error message when we encounter
  a Hive column type we can't handle yet. Bug 1392450.

- optimizer/TableDesc.*
  Method TableDesc::validateDivisionByClauseForDDL() got moved
  to CmpSeabaseDDL::validateDivisionByExprForDDL().

- optimizer/NormItemExpr.cpp
  BETWEEN transformation described above.

- optimizer/ValueDesc.cpp
  Avoid hard-codeing the "_SALT_" name and adding a comment about
  possibility to use the flag in the future.

- parser
  Lots of small changes for salt and divisioning option changes.
  Simplifying the syntax for salt options somewhat. I think the older
  syntax was so complex because it needed to record the starting and
  ending position of the divisioning clause, something we don't need
  anymore.

- regress: Adding new test

- sqlcomp/CmpDescribe.cpp: Support for describing DIVISION BY clause
  and also supporting the new WITHOUT SALT | DIVISION options
  for CREATE TABLE LIKE, which relies on the describe feature.

- sqlcomp/CmpSeabaseDDLcommon.cpp
  sqlcomp/CmpSeabaseDDL.h
  + Handling the new column flags and making sure they are not
    confused with the HBase column flags (e.g. for serialization).
  + Setting the new COLUMNS.FLAGS when writing metadata.
  + Also, writing the computed column text to the TEXT table.
  + For DROP TABLE, unconditionally deleting TEXT rows, since the
    table could contain computed columns.
  + When building ColInfoArray, check system column flags, since
    system columns can now appear at any position.
  + Add method to "bind" an item expression during DDL processing
    without going through the full binder. This replaces any column
    reference with a named NATypeToItem node, since all we really
    need is the type and the name for unparsing.
  + Method TableDesc::validateDivisionByClauseForDDL() got moved
    to CmpSeabaseDDL::validateDivisionByExprForDDL() with some minor
    adjustments, since it used to be called on a bound ItemExpr, now
    it gets called on something that came out of the parser and went
    through the DDL time "binder".

- sqlcomp/CmpSeabaseDDLindex.cpp:
  Support for CREATE INDEX ... DIVISION LIKE TABLE. If this is
  set, add the division columns in front of the index key, otherwise
  don't.

- sqlcomp/CmpSeabaseDDLtable.cpp:
  + Code to make sure column flags and column class is set and propagated.
  + Fix for bug 1385543: Now that we use the TEXT table for computed
    column text, we no longer have a length limit. This is true for both
    divisioning and salt expressions.
  + When processing the column list in seabaseCreateTable() we have a
    bit of a chicken and egg problem: We need the column list to validate
    the DIVISION BY expressions, but the DIVISION BY columns need to be part
    of the column list. So, we do this a first time without divisioning
    columns, then we add those, and produce the final list in a second
    iteration.
  + getTextFromMD method now takes a sub-id as an input parameter. That's
    the column number for computed column text.
  + read computed column text from the TEXT table. Note: This also needs
    to handle older tables where the computed column text is stored in
    the default value.

Change-Id: I7c3ebe39a950c1d01f31855bdc92cbb98e5eb275
Merge "Support for divisioning (multi-temperature data)"
Merge "Bug fixes"
Fix for fully qualified domain named host.

Change-Id: Id2ce37be414a88f7270ccdcb0ed187e533e78dcf
Merge "SQL syntax to cancel executing query, phase 1"
Merge "Fix for fully qualified domain named host."
Expected file change to address RPAD function

Change-Id: Iae75862f2e348eb620349724e096f94c7c8e6b64