Project: Beam (App Definition and Development / Streaming & Messaging)
File: lexical-structure.md

type: languages title: "Beam Calcite SQL lexical structure" aliases: /documentation/dsls/sql/lexical/ /documentation/dsls/sql/calcite/lexical/ <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> A Beam Calcite SQL statements are comprised of a series of tokens. Tokens include identifiers, quoted identifiers, literals, keywords, operators, and special characters. Tokens can be separated by whitespace (space, backspace, tab, newline) or comments. Identifiers -- Identifiers are names that are associated with columns, tables, and other database objects. Identifiers must begin with a letter or an underscore. Subsequent characters can be letters, numbers, or underscores. Quoted identifiers are identifiers enclosed by backtick (`` ` ``) characters and can contain any character, such as spaces or symbols. However, quoted identifiers cannot be empty. can only be used as identifiers if enclosed by backticks. Syntax (defined here as a regular expression): `*` Examples: ``` Customers5 _dataField1 ADGROUP ``` Invalid examples: ``` 5Customers _dataField! GROUP ``` `5Customers` begins with a number, not a letter or underscore. `_dataField!` contains the special character "!" which is not a letter, number, or underscore. `GROUP` is a reserved keyword, and therefore cannot be used as an identifier without being enclosed by backtick characters. Both identifiers and quoted identifiers are case insensitive, with some nuances. See for further details. Quoted identifiers have the same escape sequences as string literals, defined below. Literals -- A literal represents a constant value of a built-in data type. Some, but not all, data types can be expressed as literals. Both string and bytes literals must be quoted with single (`'`) quotation mark. Quoted literals: <table> <thead> <tr> <th>Literal</th> <th>Examples</th> <th>Description</th> </tr> </thead> <tbody> <tr> <td>Quoted string</td> <td><ul><li><code>'it''s'</code></li><li><code>'Title: "Boy"'</code></li></ul></td> <td>Quoted strings enclosed by single (<code>'</code>) quotes can contain unescaped double (<code>"</code>) quotes. <br />Two quotation marks (<code>''</code>) is the escape sequence.<br />Quoted strings can contain newlines.</td> </tr> </tbody> </table> Integer literals are either a sequence of decimal digits (0 through 9). Integers can be prefixed by "`+`" or "`-`" to represent positive and negative values, respectively. Examples: ``` 123 -123 ``` An integer literal is interpreted as an `BIGINT`. Syntax options: ``` DIGITS] [DIGITS].DIGITS[e[+-]DIGITS] DIGITSe[+-]DIGITS ``` `DIGITS` represents one or more decimal numbers (0 through 9) and `e` represents the exponent marker (e or E). Examples: ``` 123.456e-67 .1E4 4e2 ``` Numeric literals that contain either a decimal point or an exponent marker are presumed to be type double. Implicit coercion of floating point literals to float type is possible if the value is within the valid float range. There is no literal representation of NaN or

Array literals are a comma-separated lists of elements enclosed in square brackets prefixed with the `ARRAY` keyword. Examples: ``` ARRAY[1, 2, 3] ARRAY['x', 'y', 'xy'] ``` Syntax: ``` (elem[, elem...]) ``` where `elem` is an element in the struct. `elem` must be a literal data type, not an expression or column name. The output type is an anonymous struct type (structs are not named types) with anonymous fields with types matching the types of the input expressions. <table> <thead> <tr> <th>Example</th> <th>Output Type</th> </tr> </thead> <tbody> <tr> <td><code>(1, 2, 3)</code></td> <td><code>STRUCT&lt;BIGINT,BIGINT,BIGINT&gt;</code></td> </tr> <tr> <td><code>(1, 'abc')</code></td> <td><code>STRUCT&lt;BIGINT,STRING&gt;</code></td> </tr> </tbody> </table> Syntax: ``` DATE 'YYYY-M[M]-D[D]' ``` Date literals contain the `DATE` keyword followed by a string literal that conforms to the canonical date format, enclosed in single quotation marks. Date literals support a range between the years 1 and 9999, inclusive. Dates outside of this range are invalid. For example, the following date literal represents September 27, 2014: ``` DATE '2014-09-27' ``` String literals in canonical date format also implicitly coerce to DATE type when used where a DATE-type expression is expected. For example, in the query ``` SELECT * FROM foo WHERE date_col = "2014-09-27" ``` the string literal `"2014-09-27"` will be coerced to a date literal. Syntax: ``` TIME '[H]H:[M]M:[S]S[.DDDDDD]]' ``` TIME literals contain the `TIME` keyword and a string literal that conforms to the canonical time format, enclosed in single quotation marks. For example, the following time represents 12:30 p.m.: ``` TIME '12:30:00.45' ``` Syntax: ``` TIMESTAMP 'YYYY-[M]M-[D]D [[H]H:[M]M:[S]S[.DDDDDD]]' ``` Timestamp literals contain the `TIMESTAMP` keyword and a string literal that conforms to the canonical timestamp format, enclosed in single quotation marks. Timestamp literals support a range between the years 1 and 9999, inclusive. Timestamps outside of this range are invalid. For example, the following timestamp represents 12:30 p.m. on September 27, 2014: ``` TIMESTAMP '2014-09-27 12:30:00.45' ``` Case Sensitivity Beam SQL follows these rules for case sensitivity: <table class="table-wrapper--equal-p"> <thead> <tr> <th>Category</th> <th>Case Sensitive?</th> <th>Notes</th> </tr> </thead> <tbody> <tr> <td>Keywords</td> <td>No</td> <td></td> </tr> <tr> <td>Function names</td> <td>No</td> <td></td> </tr> <tr> <td>Table names</td> <td>Yes</td> <td></td> </tr> <tr> <td>Column names</td> <td>Yes</td> <td></td> </tr> <tr> <td>String values</td> <td>Yes</td> <td></td> </tr> <tr> <td>String comparisons</td> <td>Yes</td> <td></td> </tr> <tr> <td>Aliases within a query</td> <td>No</td> <td></td> </tr> <tr> <td>Regular expression matching</td> <td>See Notes</td> <td>Regular expression matching is case sensitive by default, unless the expression itself specifies that it should be case insensitive.</td> </tr> <tr> <td><code>LIKE</code> matching</td> <td>Yes</td> <td>&nbsp;</td> </tr> </tbody> </table> Reserved Keywords -- Keywords are a group of tokens that have special meaning in the Beam SQL language, and have the following characteristics: Keywords cannot be used as identifiers unless enclosed by backtick (\`) characters. Keywords are case insensitive. Beam SQL has the following reserved

<div class="table-container-wrapper"> <table style="width: 100%" class="table-wrapper--equal-p"> <tbody> <tr> <td> A<br /> ABS<br /> ABSOLUTE<br /> ACTION<br /> ADA<br /> ADD<br /> ADMIN<br /> AFTER<br /> ALL<br /> ALLOCATE<br /> ALLOW<br /> ALTER<br /> ALWAYS<br /> AND<br /> ANY<br /> APPLY<br /> ARE<br /> ARRAY<br /> ARRAYMAXCARDINALITY<br /> AS<br /> ASC<br /> ASENSITIVE<br /> ASSERTION<br /> ASSIGNMENT<br /> ASYMMETRIC<br /> AT<br /> ATOMIC<br /> ATTRIBUTE<br /> ATTRIBUTES<br /> AUTHORIZATION<br /> AVG<br /> BEFORE<br /> BEGIN<br /> BEGIN_FRAME<br /> BEGIN_PARTITION<br /> BERNOULLI<br /> BETWEEN<br /> BIGINT<br /> BINARY<br /> BIT<br /> BLOB<br /> BOOLEAN<br /> BOTH<br /> BREADTH<br /> BY<br /> C<br /> CALL<br /> CALLED<br /> CARDINALITY<br /> CASCADE<br /> CASCADED<br /> CASE<br /> CAST<br /> CATALOG<br /> CATALOG_NAME<br /> CEIL<br /> CEILING<br /> CENTURY<br /> CHAIN<br /> CHAR<br /> CHAR_LENGTH<br /> CHARACTER<br /> CHARACTER_LENGTH<br /> CHARACTERSETCATALOG<br /> CHARACTERSETNAME<br /> CHARACTERSETSCHEMA<br /> CHARACTERISTICS<br /> CHARACTERS<br /> CHECK<br /> CLASSIFIER<br /> CLASS_ORIGIN<br /> CLOB<br /> CLOSE<br /> COALESCE<br /> COBOL<br /> COLLATE<br /> COLLATION<br /> COLLATION_CATALOG<br /> COLLATION_NAME<br /> COLLATION_SCHEMA<br /> COLLECT<br /> COLUMN<br /> COLUMN_NAME<br /> COMMAND_FUNCTION<br /> COMMANDFUNCTIONCODE<br /> COMMENT<br /> COMMIT<br /> COMMITTED<br /> CONDITION<br /> CONDITION_NUMBER<br /> CONNECT<br /> CONNECTION<br /> CONNECTION_NAME<br /> CONSTRAINT<br /> CONSTRAINT_CATALOG<br /> CONSTRAINT_NAME<br /> CONSTRAINT_SCHEMA<br /> CONSTRAINTS<br /> CONSTRUCTOR<br /> CONTAINS<br /> CONTINUE<br /> CONVERT<br /> CORR<br /> CORRESPONDING<br /> COUNT<br /> COVAR_POP<br /> COVAR_SAMP<br /> CREATE<br /> CROSS<br /> CUBE<br /> CUME_DIST<br /> CURRENT<br /> CURRENT_CATALOG<br /> CURRENT_DATE<br /> CURRENTDEFAULTTRANSFORM_GROUP<br /> CURRENT_PATH<br /> CURRENT_ROLE<br /> CURRENT_ROW<br /> CURRENT_SCHEMA<br /> CURRENT_TIME<br /> CURRENT_TIMESTAMP<br /> CURRENTTRANSFORMGROUPFORTYPE<br /> CURRENT_USER<br /> CURSOR<br /> CURSOR_NAME<br /> CYCLE<br /> DATA<br /> DATABASE<br /> DATE<br /> DATETIMEINTERVALCODE<br /> DATETIMEINTERVALPRECISION<br /> DAY<br /> DEALLOCATE<br /> DEC<br /> DECADE<br /> DECIMAL<br /> DECLARE<br /> DEFAULT<br /> DEFAULTS<br /> DEFERRABLE<br /> DEFERRED<br /> DEFINE<br /> DEFINED<br /> DEFINER<br /> DEGREE<br /> DELETE<br /> DENSE_RANK<br /> DEPTH<br /> DEREF<br /> DERIVED<br /> DESC<br /> DESCRIBE<br /> DESCRIPTION<br /> DESCRIPTOR<br /> DETERMINISTIC<br /> DIAGNOSTICS<br /> DISALLOW<br /> DISCONNECT<br /> DISPATCH<br /> DISTINCT<br /> DOMAIN<br /> DOUBLE<br /> DOW<br /> DOY<br /> DROP<br /> DYNAMIC<br /> DYNAMIC_FUNCTION<br /> DYNAMICFUNCTIONCODE<br /> EACH<br /> ELEMENT<br /> ELSE<br /> EMPTY<br /> END<br /> END-EXEC<br /> END_FRAME<br /> END_PARTITION<br /> EPOCH<br /> EQUALS<br /> ESCAPE<br /> EVERY<br /> EXCEPT<br /> EXCEPTION<br /> EXCLUDE<br /> EXCLUDING<br /> EXEC<br /> EXECUTE<br /> EXISTS<br /> EXP<br /> EXPLAIN<br /> EXTEND<br /> EXTERNAL<br /> EXTRACT<br /> FALSE<br /> FETCH<br /> FILTER<br /> FINAL<br /> FIRST<br /> FIRST_VALUE<br /> FLOAT<br /> FLOOR<br /> FOLLOWING<br /> FOR<br /> FOREIGN<br /> FORTRAN<br /> FOUND<br /> FRAC_SECOND<br /> FRAME_ROW<br /> FREE<br /> FROM<br /> FULL<br /> FUNCTION<br /> FUSION<br /> G<br /> GENERAL<br /> GENERATED<br /> GEOMETRY<br /> GET<br /> GLOBAL<br /> GO<br /> GOTO<br /> GRANT<br /> GRANTED<br /> GROUP<br /> GROUPING<br /> GROUPS<br /> HAVING<br /> HIERARCHY<br /> HOLD<br /> HOUR<br /> IDENTITY<br /> IF<br /> IMMEDIATE<br /> IMMEDIATELY<br /> IMPLEMENTATION<br /> IMPORT<br /> IN<br /> INCLUDING<br /> INCREMENT<br /> INDICATOR<br /> INITIAL<br /> INITIALLY<br /> INNER<br /> INOUT<br /> INPUT<br /> INSENSITIVE<br /> INSERT<br /> INSTANCE<br /> INSTANTIABLE<br /> INT<br /> INTEGER<br /> INTERSECT<br /> INTERSECTION<br /> INTERVAL<br /> INTO<br /> INVOKER<br /> IS<br /> ISOLATION<br /> JAVA<br /> JOIN<br /> JSON<br /> K<br /> KEY<br /> KEY_MEMBER<br /> KEY_TYPE<br /> LABEL<br /> LAG<br /> LANGUAGE<br /> LARGE<br /> LAST<br /> LAST_VALUE<br /> LATERAL<br /> LEAD<br /> LEADING<br /> LEFT<br /> LENGTH<br /> LEVEL<br /> LIBRARY<br /> LIKE<br /> LIKE_REGEX<br /> LIMIT<br /> LN<br /> LOCAL<br /> LOCALTIME<br /> LOCALTIMESTAMP<br /> LOCATION<br /> LOCATOR<br /> LOWER<br /> M<br /> MAP<br /> MATCH<br /> MATCHED<br /> MATCHES<br /> MATCH_NUMBER<br /> MATCH_RECOGNIZE<br /> MAX<br /> MAXVALUE<br