Repository: pingcap/dumpling (go)

Issue:
dump data failed with syntax error when the entire table is generated columns
## Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.
```
dumpling -o . -P 4000 -u root -h host
```
2. What did you expect to see?
dump data successfully.
3. What did you see instead?
```
[2020/08/27 14:42:01.739 +08:00] [ERROR] [main.go:212] ["dump failed error stack info"] [error="SELECT  FROM `0247__TICASE_1512_WWSVHA`.`t` LIMIT 1: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 12 near \"FROM `0247__TICASE_1512_WWSVHA`.`t` LIMIT 1\" "] [errorVerbose="err = SELECT  FROM `0247__TICASE_1512_WWSVHA`.`t` LIMIT 1: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 12 near \"FROM `0247__TICASE_1512_WWSVHA`.`t` LIMIT 1\" \ngoroutine 1 [running]:\nruntime/debug.Stack(0x19aec00, 0xc000272f40, 0x19afec0)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9f\ngithub.com/pingcap/dumpling/v4/export.withStack(0x19aec00, 0xc000272f40, 0xc00018a000, 0xc0001a2c00)\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/export/error.go:59 +0x8f\ngithub.com/pingcap/dumpling/v4/export.GetColumnTypes(0xc000304180, 0x0, 0x0, 0xc00023ffe0, 0x18, 0x1dd1720, 0x1, 0x0, 0x0, 0x0, ...)\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/export/sql.go:263 +0x325\ngithub.com/pingcap/dumpling/v4/export.SelectAllFromTable(0xc00029e000, 0xc000304180, 0xc00023ffe0, 0x18, 0x1dd1720, 0x1, 0x1, 0xc00001a000, 0xcf, 0x0)\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/export/sql.go:141 +0x105\ngithub.com/pingcap/dumpling/v4/export.dumpTable(0x19c4cc0, 0xc0001a8cc0, 0xc00029e000, 0xc000304180, 0xc00023ffe0, 0x18, 0xc0003a1e00, 0x19c32c0, 0xc00029e000, 0xc0003f5350, ...)\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/export/dump.go:321 +0xc5\ngithub.com/pingcap/dumpling/v4/export.dumpDatabases(0x19c4cc0, 0xc0001a8cc0, 0xc00029e000, 0xc00031e0e0, 0x19c32c0, 0xc00029e000, 0x0, 0x0)\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/export/dump.go:232 +0x353\ngithub.com/pingcap/dumpling/v4/export.Dump(0x19c4d00, 0xc00018a000, 0xc00029e000, 0x0, 0x0)\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/export/dump.go:199 +0xc25\nmain.main()\n\t/Users/aytrack/gitproject/pingcap/dumpling/cmd/dumpling/main.go:210 +0x152a\n"] [stack="github.com/pingcap/dumpling/v4/log.Error\n\t/Users/aytrack/gitproject/pingcap/dumpling/v4/log/log.go:78\nmain.main\n\t/Users/aytrack/gitproject/pingcap/dumpling/cmd/dumpling/main.go:212\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"]

dump failed: SELECT  FROM `0247__TICASE_1512_WWSVHA`.`t` LIMIT 1: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your TiDB version for the right syntax to use line 1 column 12 near "FROM `0247__TICASE_1512_WWSVHA`.`t` LIMIT 1"   
```
4. Versions of the cluster

    - Dumpling version (run `dumpling -V`):

        ```
        Release version: v4.0.3-12-g8e00b82
        Git commit hash: 8e00b826f68dda6734d69d7d5c96d082b0e88ee6
        Git branch:      master
        Build timestamp: 2020-08-27 06:25:31Z
        Go version:      go version go1.15 darwin/amd64
        ```

    - Source database version (execute `SELECT version();` in a MySQL client):

        ```
        +-------------------------------------------------------------------+
        | tidb_version()                                                    |
        +-------------------------------------------------------------------+
        | Release Version: v4.0.0-beta.2-1006-g6831e487a                    |
        | Edition: Community                                                |
        | Git Commit Hash: 6831e487a84c8bde625de8c73feec9e6ec02f426         |
        | Git Branch: master                                                |
        | UTC Build Time: 2020-08-23 13:37:28                               |
        | GoVersion: go1.13                                                 |
        | Race Enabled: false                                               |
        | TiKV Min Version: v3.0.0-60965b006877ca7234adaced7890d7b029ed1306 |
        | Check Table Before Drop: false                                    |
        +-------------------------------------------------------------------+
        ```

    - Other interesting information (system version, hardware config, etc):

        >
        >

5. Operation logs
   - Please upload `dumpling.log` for Dumpling if possible
   - Other interesting logs
   
6. Configuration of the cluster and the task
   - running command for Dumpling if possible


Pull request description:
Fix: dump all generated column table
<!--
Thank you for contributing to Dumpling! Please read the [CONTRIBUTING](https://github.com/pingcap/dumpling/blob/master/CONTRIBUTING.md) document **BEFORE** filing this PR.
-->

### What problem does this PR solve? <!--add issue link with summary if exists-->
Fix #145 

### What is changed and how it works?
Handling the case where selected field is empty

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

 - Unit test

Side effects


Related changes

 
### Release note

<!-- bugfixes or new feature need a release note, must in the form of a list, such as

- support -T/--tables-list argument

or if no need to be included in the release note, just add the following line

- No release note
-->


Patch:
diff --git a/v4/export/ir_impl.go b/v4/export/ir_impl.go
index 9dbf3a9..fe89644 100644
--- a/v4/export/ir_impl.go
+++ b/v4/export/ir_impl.go
@@ -138,8 +138,8 @@ func (td *tableData) Rows() SQLRowIter {
 }
 
 func (td *tableData) SelectedField() string {
-	if td.selectedField == "*" {
-		return ""
+	if td.selectedField == "*" || td.selectedField == "" {
+		return td.selectedField
 	}
 	return fmt.Sprintf("(%s)", td.selectedField)
 }
diff --git a/v4/export/sql.go b/v4/export/sql.go
index 6bf10fa..6590344 100644
--- a/v4/export/sql.go
+++ b/v4/export/sql.go
@@ -194,7 +194,13 @@ func SelectAllFromTable(conf *Config, db *sql.Conn, database, table string) (Tab
 		return nil, err
 	}
 
-	colTypes, err := GetColumnTypes(db, selectedField, database, table)
+	var colTypes []*sql.ColumnType
+	// If all columns are generated
+	if selectedField == "" {
+		colTypes, err = GetColumnTypes(db, "*", database, table)
+	} else {
+		colTypes, err = GetColumnTypes(db, selectedField, database, table)
+	}
 	if err != nil {
 		return nil, err
 	}
@@ -204,7 +210,12 @@ func SelectAllFromTable(conf *Config, db *sql.Conn, database, table string) (Tab
 		return nil, err
 	}
 
-	query := buildSelectQuery(database, table, selectedField, buildWhereCondition(conf, ""), orderByClause)
+	queryField := selectedField
+	// If all columns are generated
+	if queryField == "" {
+		queryField = "''"
+	}
+	query := buildSelectQuery(database, table, queryField, buildWhereCondition(conf, ""), orderByClause)
 
 	return &tableData{
 		database:        database,
diff --git a/v4/export/writer_util.go b/v4/export/writer_util.go
index 6edf9fc..131f58f 100644
--- a/v4/export/writer_util.go
+++ b/v4/export/writer_util.go
@@ -159,8 +159,9 @@ func WriteInsert(pCtx context.Context, tblIR TableDataIR, w storage.Writer, file
 	)
 
 	selectedField := tblIR.SelectedField()
+
 	// if has generated column
-	if selectedField != "" {
+	if selectedField != "" && selectedField != "*" {
 		insertStatementPrefix = fmt.Sprintf("INSERT INTO %s %s VALUES\n",
 			wrapBackTicks(escapeString(tblIR.TableName())), selectedField)
 	} else {
@@ -175,13 +176,17 @@ func WriteInsert(pCtx context.Context, tblIR TableDataIR, w storage.Writer, file
 		wp.AddFileSize(insertStatementPrefixLen)
 
 		for fileRowIter.HasNext() {
-			if err = fileRowIter.Decode(row); err != nil {
-				log.Error("scanning from sql.Row failed", zap.Error(err))
-				return err
-			}
 
 			lastBfSize := bf.Len()
-			row.WriteToBuffer(bf, escapeBackSlash)
+			if selectedField != "" {
+				if err = fileRowIter.Decode(row); err != nil {
+					log.Error("scanning from sql.Row failed", zap.Error(err))
+					return err
+				}
+				row.WriteToBuffer(bf, escapeBackSlash)
+			} else {
+				bf.WriteString("()")
+			}
 			counter += 1
 			wp.AddFileSize(uint64(bf.Len()-lastBfSize) + 2) // 2 is for ",\n" and ";\n"
 
@@ -260,7 +265,7 @@ func WriteInsertInCsv(pCtx context.Context, tblIR TableDataIR, w storage.Writer,
 		err             error
 	)
 
-	if !noHeader && len(tblIR.ColumnNames()) != 0 {
+	if !noHeader && len(tblIR.ColumnNames()) != 0 && tblIR.SelectedField() != "" {
 		for i, col := range tblIR.ColumnNames() {
 			bf.Write(opt.delimiter)
 			escape([]byte(col), bf, getEscapeQuotation(escapeBackSlash, opt.delimiter))
@@ -274,13 +279,14 @@ func WriteInsertInCsv(pCtx context.Context, tblIR TableDataIR, w storage.Writer,
 	wp.currentFileSize += uint64(bf.Len())
 
 	for fileRowIter.HasNext() {
-		if err = fileRowIter.Decode(row); err != nil {
-			log.Error("scanning from sql.Row failed", zap.Error(err))
-			return err
-		}
-
 		lastBfSize := bf.Len()
-		row.WriteToBufferInCsv(bf, escapeBackSlash, opt)
+		if tblIR.SelectedField() != "" {
+			if err = fileRowIter.Decode(row); err != nil {
+				log.Error("scanning from sql.Row failed", zap.Error(err))
+				return err
+			}
+			row.WriteToBufferInCsv(bf, escapeBackSlash, opt)
+		}
 		counter += 1
 		wp.currentFileSize += uint64(bf.Len()-lastBfSize) + 1 // 1 is for "\n"
 


Test patch:
diff --git a/tests/all_generate_column/conf/diff_config.toml b/tests/all_generate_column/conf/diff_config.toml
new file mode 100644
index 0000000..1d69e86
--- /dev/null
+++ b/tests/all_generate_column/conf/diff_config.toml
@@ -0,0 +1,42 @@
+# diff Configuration.
+
+log-level = "info"
+
+chunk-size = 1000
+
+check-thread-count = 4
+
+sample-percent = 100
+
+use-rowid = false
+
+use-checksum = true
+
+fix-sql-file = "fix.sql"
+
+# tables need to check.
+[[check-tables]]
+schema = "all_generate_column"
+tables = ["~t.*"]
+
+[[table-config]]
+schema = "all_generate_column"
+table = "t"
+
+[[table-config.source-tables]]
+instance-id = "source-1"
+schema = "all_generate_column"
+table  = "t"
+
+[[source-db]]
+host = "127.0.0.1"
+port = 3306
+user = "root"
+password = ""
+instance-id = "source-1"
+
+[target-db]
+host = "127.0.0.1"
+port = 4000
+user = "root"
+password = ""
diff --git a/tests/all_generate_column/conf/lightning.toml b/tests/all_generate_column/conf/lightning.toml
new file mode 100644
index 0000000..1dada67
--- /dev/null
+++ b/tests/all_generate_column/conf/lightning.toml
@@ -0,0 +1,20 @@
+### tidb-lightning config
+
+[lightning]
+server-mode = false
+level = "error"
+check-requirements = false
+
+[tikv-importer]
+backend="tidb"
+on-duplicate = "error"
+
+[mydumper]
+data-source-dir = "/tmp/dumpling_test_result/sql_res.all_generate_column"
+
+[tidb]
+host = "127.0.0.1"
+port = 4000
+user = "root"
+password = ""
+status-port = 10080
diff --git a/tests/all_generate_column/run.sh b/tests/all_generate_column/run.sh
new file mode 100644
index 0000000..a9d7c6a
--- /dev/null
+++ b/tests/all_generate_column/run.sh
@@ -0,0 +1,37 @@
+#!/bin/sh
+
+set -eu
+cur=$(cd `dirname $0`; pwd)
+
+DB_NAME="all_generate_column"
+TABLE_NAME="t"
+
+# drop database on tidb
+export DUMPLING_TEST_PORT=4000
+run_sql "drop database if exists $DB_NAME;"
+
+# drop database on mysql
+export DUMPLING_TEST_PORT=3306
+run_sql "drop database if exists $DB_NAME;"
+
+# build data on mysql
+run_sql "create database $DB_NAME;"
+
+# build data with generate column full_name
+run_sql "create table $DB_NAME.$TABLE_NAME(a int as (1), b int as (2)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;"
+
+# insert 100 records
+run_sql "insert into $DB_NAME.$TABLE_NAME values $(seq -s, 100 | sed 's/,*$//g' | sed "s/[0-9]*/()/g");"
+
+# dumping
+export DUMPLING_TEST_DATABASE=$DB_NAME
+run_dumpling
+
+cat "$cur/conf/lightning.toml"
+# use lightning import data to tidb
+run_lightning $cur/conf/lightning.toml
+
+# check mysql and tidb data
+check_sync_diff $cur/conf/diff_config.toml
+
+
diff --git a/v4/export/test_util.go b/v4/export/test_util.go
index 58813c7..d7300aa 100644
--- a/v4/export/test_util.go
+++ b/v4/export/test_util.go
@@ -115,9 +115,6 @@ func (m *mockTableIR) ColumnNames() []string {
 }
 
 func (m *mockTableIR) SelectedField() string {
-	if m.selectedField == "*" {
-		return ""
-	}
 	return m.selectedField
 }