Project: Flink (App Definition and Development / Streaming & Messaging)
File: determinism.md

title: "Determinism in Continuous Queries" weight: 2 type: docs aliases: /zh/dev/table/streaming/determinism.html <!-- Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> This article is about: What is determinism? Is all batch processing deterministic? Two examples of batch queries with non-deterministic results Non-determinism in batch processing Determinism in streaming processing Non-determinism in streaming Non-deterministic update in streaming How to eliminate the impact of non-deterministic update in streaming queries Quoting the SQL standard's description of determinism: 'an operation is deterministic if that operation assuredly computes identical results when repeated with identical input values'. In a classic batch scenario, repeated execution of the same query for a given bounded data set will yield consistent results, which is the most intuitive understanding of determinism. In practice, however, the same query does not always return consistent results on a batch process either, looking at two example queries. For example, there is a newly created website click log table: ```sql CREATE TABLE clicks ( uid VARCHAR(128), cTime TIMESTAMP(3), url VARCHAR(256) ) ``` Some new data were written: ``` ++++ | uid | cTime | url | ++++ | Mary | 2022-08-22 12:00:01 | /home | | Bob | 2022-08-22 12:00:01 | /home | | Mary | 2022-08-22 12:00:05 | /prod?id=1 | | Liz | 2022-08-22 12:01:00 | /home | | Mary | 2022-08-22 12:01:30 | /cart | | Bob | 2022-08-22 12:01:35 | /prod?id=3 | ++++ ``` Query 1 applies a time filter to the log table and wants to filter out the last 2 minutes logs: ```sql SELECT * FROM clicks WHERE cTime BETWEEN TIMESTAMPADD(MINUTE, -2, CURRENTTIMESTAMP) AND CURRENTTIMESTAMP; ``` When the query was submitted at '2022-08-22 12:02:00', it returned all 6 rows in the table, and when it was executed again a minute later, at '2022-08-22 12:03:00', it returned only 3 items: ``` ++++ | uid | cTime | url | ++++ | Liz | 2022-08-22 12:01:00 | /home | | Mary | 2022-08-22 12:01:30 | /cart | | Bob | 2022-08-22 12:01:35 | /prod?id=3 | ++++ ``` Query 2 wants to add a unique identifier to each returned record (since the `clicks` table does not have a primary key) ```sql SELECT UUID() AS uuid, * FROM clicks LIMIT 3; ``` Executing this query twice in a row generates a different `uuid` identifier for each row ``` -- first execution +--++++ | uuid | uid | cTime | url | +--++++ | aaaa4894-16d4-44d0-a763-03f... | Mary | 2022-08-22 12:00:01 | /home | | ed26fd46-960e-4228-aaf2-0aa... | Bob | 2022-08-22 12:00:01 | /home | | 1886afc7-dfc6-4b20-a881-b0e... | Mary | 2022-08-22 12:00:05 | /prod?id=1 | +--++++ -- second execution +--++++ | uuid | uid | cTime | url | +--++++ | 95f7301f-bcf2-4b6f-9cf3-1ea... | Mary | 2022-08-22 12:00:01 | /home | |

| Bob | 2022-08-22 12:00:01 | /home | | f24456d3-e942-43d1-a00f-fdb... | Mary | 2022-08-22 12:00:05 | /prod?id=1 | +--++++ ``` The non-determinism in batch processing mainly caused by the non-deterministic functions, as the two query examples above, where the built-in functions `CURRENT_TIMESTAMP` and `UUID()` actually behave differently in batch processing, continue with a query example: ```sql SELECT CURRENT_TIMESTAMP, * FROM clicks; ``` `CURRENT_TIMESTAMP` is the same value on all records returned ``` +-++++ | CURRENT_TIMESTAMP | uid | cTime | url | +-++++ | 2022-08-23 17:25:46.831 | Mary | 2022-08-22 12:00:01 | /home | | 2022-08-23 17:25:46.831 | Bob | 2022-08-22 12:00:01 | /home | | 2022-08-23 17:25:46.831 | Mary | 2022-08-22 12:00:05 | /prod?id=1 | | 2022-08-23 17:25:46.831 | Liz | 2022-08-22 12:01:00 | /home | | 2022-08-23 17:25:46.831 | Mary | 2022-08-22 12:01:30 | /cart | | 2022-08-23 17:25:46.831 | Bob | 2022-08-22 12:01:35 | /prod?id=3 | +-++++ ``` This difference is due to the fact that Flink inherits the definition of functions from Apache Calcite, where there are two types of functions other than deterministic function (non-deterministic functions) and dynamic function (dynamic functions, built-in dynamic functions are mainly temporal functions). The non-deterministic functions are executed at runtime (in clusters, evaluated per record), while the dynamic functions determine the corresponding values only when the query plan is generated(not executed at runtime, different values are obtained at different times, but the same values are obtained at the same execution). For more information see . A core difference between streaming and batch is the unboundedness of the data. Flink SQL abstracts streaming processing as the . So the dynamic function in the batch query example is equivalent to a non-deterministic function in a streaming processing(where logically every change in the base table triggers the query to be executed). If the `clicks` log table in the example is from a Kafka topic that is continuously written, the same query in stream mode will return `CURRENT_TIMESTAMP` that will change over time ```sql SELECT CURRENT_TIMESTAMP, * FROM clicks; ``` e.g, ``` +-++++ | CURRENT_TIMESTAMP | uid | cTime | url | +-++++ | 2022-08-23 17:25:46.831 | Mary | 2022-08-22 12:00:01 | /home | | 2022-08-23 17:25:47.001 | Bob | 2022-08-22 12:00:01 | /home | | 2022-08-23 17:25:47.310 | Mary | 2022-08-22 12:00:05 | /prod?id=1 | +-++++ ``` In addition to the non-deterministic function, other factors that may generate non-determinism are mainly: non-deterministic back read of source connector query based on clear internal state data based on For Flink SQL, the determinism provided is limited to the computation only, because it does not store user data itself (here it is necessary to distinguish between the managed internal state in streaming mode and the user data itself), so the Source connector's implementation that cannot provide deterministic back read will bring non-determinism of the input data, which may produce non-deterministic results. Common examples are inconsistent data for multiple reads from a same offset, or requests for data that no longer exists because of the retention time(e.g., the requested data beyond the configured ttl of Kafka topic). Unlike event time, processing time is based on the machine's local time, and this processing does not provide determinism. Related operations that rely on the time attribute include , , ,

Another typical operation is , which is semantically similar to Temporal Join based on processing time, where non-determinism arises when the accessed external table is changing over time. Because of the unbounded nature of streaming processing, the internal state data maintained by long-running streaming queries in operations such as and (non-windowed aggregation) may continuously get larger. Setting a state TTL to clean up internal state data is often a necessary compromise, but this may also make the computation results non-deterministic. The impact of the non-determinism on different queries is different, for some queries it just produces non-deterministic results (the query works fine, but multiple runs fail to produce consistent results), while some queries can have more serious effects, such as incorrect results or runtime errors. The main reason for the latter one is 'non-deterministic update'. Flink SQL implements a complete incremental update mechanism based on the abstraction. All operations that need to generate incremental messages maintain complete internal state data, and the operation of the entire query pipeline(include the complete dag from source to sink operators) relies on the guarantee of correct delivery of update messages between operators, which can be broken by non-determinism leading to errors. What is the 'Non-deterministic Update'(NDU)? Update messages(changelog) may contain kinds of message types: Insert (I), Delete (D), UpdateBefore (UB) and UpdateAfter (UA). There's no NDU problem in an insert-only changelog pipeline. When there is an update message (containing at least one message D, UB, UA in addition to I), the update key of the message (which can be regarded as the primary key of the changelog) is deduced from the query: when the update key can be deduced, the operators in the pipeline maintains the internal state by the update key when the update key cannot be deduced (it is possible that the primary key is not defined in the or Sink table, or some operations cannot be deduced from the semantics of the query). All operators maintaining internal state can only process update (D/UB/UA) messages through complete rows, sink nodes work in retract mode when no primary key is defined, and delete operations are performed by complete rows. Therefore, in the update-by-row mode, all the update messages received by the operators that need to maintain the state cannot be interfered by the non-deterministic column values, otherwise it will cause NDU problems resulting in computation errors. On query pipeline with update messages and cannot derive the update key, the following three points are the most important sources of NDU problems: Non-deterministic functions(include scalar, table, aggregate functions, builtin or custom ones) LookupJoin on an evolving source carries metadata fields(system columns, not belongs to the entity row itself) Note: Exceptions caused by cleaning internal state data based on TTL will be discussed separately as a runtime fault-tolerant handling strategy (). The NDU problem in streaming queries is usually not intuitive, and the risk of the problem may arise from a small change in a complex query. Since

Flink SQL () introduces an experimental NDU handling mechanism When `TRY_RESOLVE` mode is enabled, it will check whether there is NDU problem in the streaming query and try to eliminate the NDU problem generated by Lookup Join (internal materialization will be added), if there are still factors in point 1 or 3 above that cannot be eliminated automatically, Flink SQL will give as detailed error messages to prompt the user to adjust the SQL to avoid introducing non-determinism(considering the high cost and complexity of operators brought by materialization, there is no corresponding automatic resolution mechanism supported yet). Enable `TRY_RESOLVE` mode before running the streaming query, when you check that there is an unsolvable NDU problem in the query, try to modify the SQL according to the error prompt to avoid the problem proactively. A real case from : ```sql INSERT INTO tjoinsink SELECT o.orderid, o.ordername, l.logisticsid, l.logisticstarget, l.logistics_source, now() FROM t_order AS o LEFT JOIN tlogistics AS l ON ord.orderid=logistics.order_id ``` The execution plan generated by default will run with a runtime error, when `TRY_RESOLVE` mode is enabled, the following error will be given: ``` org.apache.flink.table.api.TableException: The column(s): logistics_time(generated by non-deterministic function: NOW ) can not satisfy the determinism requirement for correctly processing update message(changelogMode contains not only insert I).... Please consider removing these non-deterministic columns or making them deterministic. related rel plan: Calc(select=[CAST(orderid AS INTEGER) AS orderid, ordername, logisticsid, logisticstarget, logisticssource, CAST(NOW() AS TIMESTAMP(6)) AS logistics_time], changelogMode=[I,UB,UA,D], upsertKeys=[]) +- Join(joinType=[LeftOuterJoin], where=[=(orderid, orderid0)], select=[orderid, ordername, logisticsid, logisticstarget, logisticssource, orderid0], leftInputSpec=[JoinKeyContainsUniqueKey], rightInputSpec=[HasUniqueKey], changelogMode=[I,UB,UA,D], upsertKeys=[]) :- Exchange(distribution=[hash[order_id]], changelogMode=[I,UB,UA,D], upsertKeys=[{0}]) : +- TableSourceScan(table=[[defaultcatalog, defaultdatabase, torder, project=[orderid, ordername], metadata=[]]], fields=[orderid, order_name], changelogMode=[I,UB,UA,D], upsertKeys=[{0}]) +- Exchange(distribution=[hash[order_id]], changelogMode=[I,UB,UA,D], upsertKeys=[]) +- TableSourceScan(table=[[defaultcatalog, defaultdatabase, tlogistics, project=[logisticsid, logisticstarget, logisticssource, orderid], metadata=[]]], fields=[logisticsid, logisticstarget, logisticssource, order_id], changelogMode=[I,UB,UA,D], upsertKeys=[{0}]) ``` Follow the error prompt, remove the `now()` function or use other deterministic functions instead (or use the time field in the `order` table), you can eliminate the above NDU problem and avoid runtime errors. When using Lookup Join, try to declare the primary key(if exists) Lookup source table with primary key definitions can in many cases prevent Flink SQL from deriving update keys, thus saving the high materialization cost The following two examples shows why users are encouraged to declare primary keys for lookup source tables ```sql insert into sinkwithpk select t1.a, t1.b, t2.c from ( select *, proctime() proctime from cdc ) t1 join dimwithpk for system_time as of t1.proctime as t2 on t1.a = t2.a -- plan: the upsertKey of left stream is reserved when lookup table with a pk definition and use it as lookup key, so that the high cost materialization can be omitted. Sink(table=[defaultcatalog.defaultdatabase.sinkwithpk], fields=[a, b, c]) +- Calc(select=[a, b, c]) +- LookupJoin(table=[defaultcatalog.defaultdatabase.dimwithpk], joinType=[InnerJoin], lookup=[a=a], select=[a, b, a, c]) +- DropUpdateBefore +- TableSourceScan(table=[[defaultcatalog, defaultdatabase, cdc, project=[a, b], metadata=[]]], fields=[a, b]) ``` ```sql insert into sinkwithpk select t1.a, t1.b, t2.c from ( select *, proctime() proctime from cdc ) t1 join dimwithoutpk for system_time as of t1.proctime as t2 on t1.a = t2.a -- execution plan when `TRYRESOLVE` is enabled(may encounter errors at runtime when `TRYRESOLVE` mode is not enabled): Sink(table=[defaultcatalog.defaultdatabase.sinkwithpk], fields=[a, b, c], upsertMaterialize=[true]) +- Calc(select=[a, b, c]) +- LookupJoin(table=[defaultcatalog.defaultdatabase.dimwithoutpk], joinType=[InnerJoin], lookup=[a=a], select=[a, b, a, c], upsertMaterialize=[true]) +- TableSourceScan(table=[[defaultcatalog, defaultdatabase, cdc, project=[a, b], metadata=[]]], fields=[a, b]) ``` Though the second case can be solved by adding materialization if `TRY_RESOLVE` is enabled, but the cost is very high, there will be two more costly materialization compared to the one with primary key. When the lookup source table accessed by Lookup Join is static, the `TRY_RESOLVE` mode may not be enabled When Lookup Join accesses a static lookup