Issue 1:
Release 5.1.1

Hi community,

After the release of 5.1.0 version, we have received a lot of feedback

from users and developers. Now we plan to release the ShardingSphere 5.1.1

version this week, which has made a lot of optimization and improvement for

these feedback.

Please refer to this release notes to confirm whether it contains the

features you expect. If I miss anything, please remind me ASAP. If there

are any suggestions, please feel free to tell us.

Mail: https://lists.apache.org/thread/o9ys6db6mdnwlx0osd4nflw1rwbb3086

Release Notes

New Feature

Kernel: support alter materialized view for PostgreSQL

Kernel: support declare for PostgreSQL

Kernel: support discard for PostgreSQL

Kernel: Add mode to parser to support $$ in PostgreSQL

Kernel: Support MySQL create tablespace statement parse

Scaling: Implement stop source writing and restore source writing

Scaling: Support partial tables scale-out

DistSQL: New DistSQL syntax: SHOW UNUSED RESOURCES

Mode: Added persistent XA Recovery Id to Governance Center

Mode: Database discovery adds delayed master-slave delay function

Distributed Transaction: Add savepoint support for ShardingSphere proxy

Distributed Transaction: Support auto rollback when report exception in transaction block for PostgreSQL and openGauss

Distributed Transaction: Make it is easy to use with Narayana

Distributed Transaction: Add savepoint support for ShardingSphere-JDBC

Enhancement

Kernel: Refactor kernel to improve performance

Proxy: Reduce Docker image size of ShardingSphere-Proxy

Proxy: ShardingSphere-Proxy supports set names statements

Proxy: ShardingSphere-Proxy MySQL supports multi statements

Scaling: Only one proxy node could do data consistency check in proxy cluster

Scaling: Replace scaling input and output config fields type from int to Integer

Scaling: Update MySQL checksum SQL

Scaling: Improve scaling job progress deletion in reset and progress check before starting job

Scaling: Improve FinishCheckJob data consistency check when target tables already have the same data as source tables

Scaling: Break scaling job ASAP when there is unsupported table since primary key

Scaling: Reuse ClusterPersistRepository of proxy in PipelineAPIFactory

Scaling: Update jobId generation algorithm, and make it support idempotency

DistSQL: Support configuration data type and length when CREATE/ALTER ENCRYPT RULE

DistSQL: Unify the display results of SHOW ALL VARIABLES and SHOW VARIABLE

DistSQL: Remove the effect of binding order when DROP BINDING TABLE RULES

DistSQL: Add column mode_type in the result of SHOW INSTANCE LIST

DistSQL: Add validation to the mode when ENABLE/DISABLE INSTANCE

DistSQL: Check if the rule is in used when DROP READWRITE_SPLITTING RULE

DistSQL: Check duplicate resource names when CREATE READWRITE_SPLITTING RULE

DistSQL: Add column delay_time to the result of SHOW READWRITE_SPLITTING READ RESOURCES

DistSQL: Support IF EXISTS when DROP RULE

DistSQL: Optimize the prompt information of connection failure when ADD/ALTER RESOURCE

Mode: Add schema-level global distributed locks

Mode: Add schema version number to support batch execution of DistSQL

Mode: Persistent metadata optimization in cluster mode

Mode: The database discovery add the schemaName identifier when create a JOB

Refactor

Kernel: Refactor test case for encrypt

Kernel: Refactor metadata to support PostgreSQL database and schema

Scaling: Remove HikariCP dependency in pipeline modules

Mode: Refactor governance center storage node structure

Mode: Refactor governance center meta data structure

Mode: Adjust the database discovery MGR module to MySQL module

Bug Fix

Kernel: Fix function with no parameter

Kernel: Fix InsertValueContext.getValue cast exception

Kernel: Fix aggregate distinct column error

Kernel: Fix NPE when rewrite parameter with schema

Kernel: Fix NPE caused by GeneratedKeysResultSet not return columnName in read-write splitting

Kernel: Fix show tables statement loses part of the single table

Kernel: Fix ModShardingAlgorithm wrong route result when exist same suffix table

Kernel: Fix sql parse error when contains key in assignment clause and optimize index parse

Kernel: Fix NumberFormatException when sharding algorithm config number props

Kernel: Fix wrong metadata when config single dataSource for read-write splitting

Kernel: Fix statement close exception when use BatchPreparedStatementExecutor

Kernel: Fix rewrite lowercase logic when sql contains shorthand projection

Kernel: Fix NullPointerException when start up proxy with memory mode

Proxy: Fix literals may be replaced by mistake in PostgreSQL/openGauss protocol

Proxy: Fix ShardingSphere-Proxy PostgreSQL with multi-schema cannot be connected by PostgreSQL JDBC Driver 42.3.x

Proxy: Fix timestamp nanos inaccurate in ShardingSphere-Proxy MySQL

Proxy: Complete ShardingSphere-Proxy PostgreSQL codec for numeric in binary format

Proxy: Potential performance issue and risk of OOM in ShardingSphere-JDBC

Proxy: Fix Operation not allowed after ResultSet closed occasionally happens in ShardingSphere-Proxy MySQL

Proxy: Fix NPE causes by ShardingSphere-JDBC executeBatch without addBatch

Scaling: Fix failed or stopped job could not be started by DistSQL except restarting proxy

DistSQL: Fix parsing exception for inline expression when CREATE SHARDING TABLE RULE

DistSQL: Fix parsing exception when password is keyword password in ADD RESOURCE statement

Mode: Fixed loss of compute nodes due to ZooKeeper session timeout

Mode: Fixed the case of the table name in the governance center

Mode: DistSQL enable disable instance refresh in-memory compute node status

Mode: Fixed database discovery unable to create Rule through DistSQL

Issue 2:
Bug Report

[ERROR] 2021-07-05 15:43:33.881 [ShardingSphere-Scaling-execute-5] o.a.s.s.c.e.i.AbstractImporter - flush failed 0/3 times

.

java.sql.BatchUpdateException: 2Unknown exception: [INSERT INTO .... ON DUPLICATE KEY UPDATE can not support update for sh

arding column.]

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)

at com.mysql.jdbc.Util.getInstance(Util.java:408)

at com.mysql.jdbc.SQLError.createBatchUpdateException(SQLError.java:1163)

at com.mysql.jdbc.PreparedStatement.executeBatchedInserts(PreparedStatement.java:1632)

at com.mysql.jdbc.PreparedStatement.executeBatchInternal(PreparedStatement.java:1298)

at com.mysql.jdbc.StatementImpl.executeBatch(StatementImpl.java:970)

at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128)

at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.executeBatchInsert(AbstractImporter.j

ava:167)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.doFlush(AbstractImporter.java:142)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.tryFlush(AbstractImporter.java:127)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.flushInternal(AbstractImporter.java:1

18)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.lambda$flush$2(AbstractImporter.java:

109)

at java.util.ArrayList.forEach(ArrayList.java:1259)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.flush(AbstractImporter.java:104)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.write(AbstractImporter.java:89)

at org.apache.shardingsphere.scaling.core.executor.importer.AbstractImporter.start(AbstractImporter.java:81)

at org.apache.shardingsphere.scaling.core.executor.AbstractScalingExecutor.run(AbstractScalingExecutor.java:47)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(T

rustedListenableFutureTask.java:125)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:69)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.sql.SQLException: 2Unknown exception: [INSERT INTO .... ON DUPLICATE KEY UPDATE can not support update for

sharding column.]

at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3978)

at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3914)

at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2530)

at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2683)

at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2495)

at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1903)

at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2124)

at com.mysql.jdbc.PreparedStatement.executeUpdateInternal(PreparedStatement.java:2058)

at com.mysql.jdbc.PreparedStatement.executeLargeUpdate(PreparedStatement.java:5158)

at com.mysql.jdbc.PreparedStatement.executeBatchedInserts(PreparedStatement.java:1593)

... 21 common frames omitted

Which version of ShardingSphere did you use?

apache-shardingsphere-5.0.0-beta-shardingsphere-scaling-bin.tar.gz

Which project did you use? ShardingSphere-JDBC or ShardingSphere-Proxy?

I use scaling5.0 migration data to ShardingSphere-Proxy4.1.0

Expected behavior

Actual behavior

Reason analyze (If you can)

Steps to reproduce the behavior, such as: SQL to execute, sharding rule configuration, when exception occur etc.

Example codes for reproduce this issue (such as a github link).