Repository: pingcap/dumpling
Pull request #302 (2021-07-06 09:28:07)

export: don't +1 TSO when set GC safepoint (#290)
This is an automated cherry-pick of #290

<!--
Thank you for contributing to Dumpling! Please read the [CONTRIBUTING](https://github.com/pingcap/dumpling/blob/master/CONTRIBUTING.md) document **BEFORE** filing this PR.

If you want to open the **Challenge Program** pull request, please use the following template:
https://raw.githubusercontent.com/pingcap/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md

You can use it with query parameters: https://github.com/pingcap/dumpling/compare/master...${you branch}?template=challenge-program.md
-->

### What problem does this PR solve? <!--add issue link with summary if exists-->

this error is hard to reproduce, so I added a 10 minutes sleep before any dump indeed occurs. The error may be sliently different but I think the root cause is same . Here's the log

```
[2021/06/03 19:20:30.262 +08:00] [INFO] [gc_worker.go:1531] ["[gc worker] sent safe point to PD"] [uuid=5e7451a0ff80001] ["safe point"=425385803120640001]
...
[2021/06/03 19:22:21.578 +08:00] [INFO] [set.go:312] ["load snapshot info schema"] [conn=5813] [SnapshotTS=425385803120640000]
...
[2021/06/03 19:22:21.579 +08:00] [INFO] [conn.go:812] ["command dispatched failed"] [conn=5813] [connInfo="id:5813, addr:127.0.0.1:62687 status:10, collation:utf8mb4_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="SET tidb_snapshot='2021-06-03 19:01:00'"] [txn_mode=PESSIMISTIC] [err="[tikv:9006]GC life time is shorter than transaction duration, transaction starts at 2021-06-03 19:01:00 +0800 CST, GC safe point is 2021-06-03 19:01:00 +0800 CST\ngithub.com/pingcap/errors.AddStack\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20201126102027-b0a155152ca3/normalize.go:156\ngithub.com/pingcap/tidb/store/tikv.(*KVStore).CheckVisibility\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/store/tikv/kv.go:104\ngithub.com/pingcap/tidb/store/tikv.(*tikvSnapshot).Get\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/store/tikv/snapshot.go:387\ngithub.com/pingcap/tidb/structure.(*TxStructure).Get\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/structure/string.go:36\ngithub.com/pingcap/tidb/structure.(*TxStructure).GetInt64\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/structure/string.go:45\ngithub.com/pingcap/tidb/meta.(*Meta).GetSchemaVersion\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/meta/meta.go:260\ngithub.com/pingcap/tidb/domain.(*Domain).loadInfoSchema\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/domain/domain.go:102\ngithub.com/pingcap/tidb/domain.(*Domain).GetSnapshotInfoSchema\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/domain/domain.go:316\ngithub.com/pingcap/tidb/executor.(*SetExecutor).loadSnapshotInfoSchemaIfNeeded\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/set.go:314\ngithub.com/pingcap/tidb/executor.(*SetExecutor).setSysVariable\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/set.go:195\ngithub.com/pingcap/tidb/executor.(*SetExecutor).Next\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/set.go:108\ngithub.com/pingcap/tidb/executor.Next\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/executor.go:277\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelayExecutor\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:537\ngithub.com/pingcap/tidb/executor.(*ExecStmt).handleNoDelay\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:418\ngithub.com/pingcap/tidb/executor.(*ExecStmt).Exec\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/executor/adapter.go:368\ngithub.com/pingcap/tidb/session.runStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/session/session.go:1531\ngithub.com/pingcap/tidb/session.(*session).ExecuteStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/session/session.go:1426\ngithub.com/pingcap/tidb/server.(*TiDBContext).ExecuteStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/driver_tidb.go:218\ngithub.com/pingcap/tidb/server.(*clientConn).handleStmt\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:1630\ngithub.com/pingcap/tidb/server.(*clientConn).handleQuery\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:1503\ngithub.com/pingcap/tidb/server.(*clientConn).dispatch\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:1037\ngithub.com/pingcap/tidb/server.(*clientConn).Run\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/conn.go:795\ngithub.com/pingcap/tidb/server.(*Server).onConn\n\t/Users/pingcap/workspace/optimization-build-tidb-darwin-amd/go/src/github.com/pingcap/tidb/server/server.go:477\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357"]
```

### What is changed and how it works?

as title

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

 - Manual test (TODO)

Side effects


Related changes

 - Need to cherry-pick to the release branch
 
### Release note

<!-- bugfixes or new feature need a release note, must in the form of a list, such as

- fix a BUG that may encounter "Error 9006, GC life time is shorter than transaction duration"


ti-chi-bot: @ti-chi-bot: This cherry pick PR is for a release branch and has not yet been approved by release team.
Adding the `do-not-merge/cherry-pick-not-approved` label.

To merge this cherry pick, it must first be approved by the collaborators.

**AFTER** it has been approved by collaborators, please ping the release team in a comment to request a cherry pick review.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
ti-chi-bot: [REVIEW NOTIFICATION]

This pull request has not been approved.


To complete the [pull request process](https://book.prow.tidb.io/#/en/workflows/pr), please ask the reviewers in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/302/owners) to review by filling `/cc @reviewer` in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/302/owners) by filling  `/assign @committer` in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found [here](https://prow.tidb.io/command-help?repo=pingcap%2Fdumpling).

<details>

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.
</details>

<!--Review Notification Identifier-->
lichunzhu: /merge
ti-chi-bot: @lichunzhu: `/merge` in this pull request requires 1 approval(s).

<details>

In response to [this](https://github.com/pingcap/dumpling/pull/302#issuecomment-874615672):

>/merge


Instructions for interacting with me using PR comments are available [here](https://prow.tidb.io/command-help).  If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
</details>
ti-chi-bot: [REVIEW NOTIFICATION]

This pull request has been approved by:

- lichunzhu




To complete the [pull request process](https://book.prow.tidb.io/#/en/workflows/pr), please ask the reviewers in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/302/owners) to review by filling `/cc @reviewer` in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/302/owners) by filling  `/assign @committer` in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found [here](https://prow.tidb.io/command-help?repo=pingcap%2Fdumpling).

<details>

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.
</details>

<!--Review Notification Identifier-->
lichunzhu: /merge
ti-chi-bot: This pull request has been accepted and is ready to merge. <details>Commit hash: 9a6fc2610c1b77ecd97c181ab9b41a04316b4340</details>

Patch:
diff --git a/v4/export/sql.go b/v4/export/sql.go
index 655d9c2..d40b4a1 100644
--- a/v4/export/sql.go
+++ b/v4/export/sql.go
@@ -967,7 +967,7 @@ func parseSnapshotToTSO(pool *sql.DB, snapshot string) (uint64, error) {
 	if !tso.Valid {
 		return 0, errors.Errorf("snapshot %s format not supported. please use tso or '2006-01-02 15:04:05' format time", snapshot)
 	}
-	return (uint64(tso.Int64)<<18)*1000 + 1, nil
+	return (uint64(tso.Int64) << 18) * 1000, nil
 }
 
 func buildWhereCondition(conf *Config, where string) string {


Test patch:
diff --git a/v4/export/sql_test.go b/v4/export/sql_test.go
index 20d5e49..4f12ed8 100644
--- a/v4/export/sql_test.go
+++ b/v4/export/sql_test.go
@@ -316,7 +316,7 @@ func (s *testSQLSuite) TestParseSnapshotToTSO(c *C) {
 		WillReturnRows(sqlmock.NewRows([]string{`unix_timestamp("2020/07/18 20:31:50")`}).AddRow(1595075510))
 	tso, err := parseSnapshotToTSO(db, snapshot)
 	c.Assert(err, IsNil)
-	c.Assert(tso, Equals, (unixTimeStamp<<18)*1000+1)
+	c.Assert(tso, Equals, (unixTimeStamp<<18)*1000)
 	c.Assert(mock.ExpectationsWereMet(), IsNil)
 
 	// generate columns not valid snapshot