Repository: pingcap/dumpling
Pull request #270 (2021-04-13 08:35:02)

log: replace pingcap global logger when start dumpling binary
<!--
Thank you for contributing to Dumpling! Please read the [CONTRIBUTING](https://github.com/pingcap/dumpling/blob/master/CONTRIBUTING.md) document **BEFORE** filing this PR.

If you want to open the **Challenge Program** pull request, please use the following template:
https://raw.githubusercontent.com/pingcap/.github/master/.github/PULL_REQUEST_TEMPLATE/challenge-program.md

You can use it with query parameters: https://github.com/pingcap/dumpling/compare/master...${you branch}?template=challenge-program.md
-->

### What problem does this PR solve? <!--add issue link with summary if exists-->
close https://github.com/pingcap/dumpling/issues/269

### What is changed and how it works?
Replace `pingcap/log`'s global logger with dumpling's to make sure pd client's log is printed to dumpling's log for dumpling binary.

### Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

 - Unit test
 - Integration test
 - Manual test (add detailed scripts or steps below)
1. Run `./bin/dumpling -P 4000 -h 127.0.0.1 -u root -L dumpling.log --loglevel fatal`:
  no log is printed.
2. Run `./bin/dumpling -P 4000 -h 127.0.0.1 -u root -L dumpling.log`:
  All logs are printed to `dumpling.log`. 
```log
[2021/04/13 16:31:56.847 +08:00] [INFO] [versions.go:55] ["Welcome to dumpling"] ["Release Version"=v5.0.0-nightly-12-g5228322] ["Git Commit Hash"=52283228cf6ca2c8adfd2147e3f7de074298a8d3] ["Git Branch"=replacePDClientLogger] ["Build timestamp"="2021-04-13 08:12:34"] ["Go Version"="go version go1.13.4 darwin/amd64"]
[2021/04/13 16:31:56.850 +08:00] [INFO] [config.go:599] ["detect server type"] [type=TiDB]
[2021/04/13 16:31:56.850 +08:00] [INFO] [config.go:618] ["detect server version"] [version=4.0.11]
[2021/04/13 16:31:56.854 +08:00] [INFO] [client.go:193] ["[pd] create pd client with endpoints"] [pd-address="[127.0.0.1:2379]"]
[2021/04/13 16:31:56.856 +08:00] [INFO] [base_client.go:308] ["[pd] switch leader"] [new-leader=http://127.0.0.1:2379] [old-leader=]
[2021/04/13 16:31:56.856 +08:00] [INFO] [base_client.go:112] ["[pd] init cluster id"] [cluster-id=6950550212317239842]
[2021/04/13 16:31:56.856 +08:00] [INFO] [dump.go:936] ["generate dumpling gc safePoint id"] [id=dumpling_1618302716856451000]
[2021/04/13 16:31:56.880 +08:00] [INFO] [dump.go:83] ["begin to run Dump"] [conf="{\"s3\":{\"endpoint\":\"\",\"region\":\"\",\"storage-class\":\"\",\"sse\":\"\",\"sse-kms-key-id\":\"\",\"acl\":\"\",\"access-key\":\"\",\"secret-access-key\":\"\",\"provider\":\"\",\"force-path-style\":true,\"use-accelerate-endpoint\":false},\"gcs\":{\"endpoint\":\"\",\"storage-class\":\"\",\"predefined-acl\":\"\",\"credentials-file\":\"\"},\"AllowCleartextPasswords\":false,\"SortByPk\":true,\"NoViews\":true,\"NoHeader\":false,\"NoSchemas\":false,\"NoData\":false,\"CompleteInsert\":false,\"TransactionalConsistency\":true,\"EscapeBackslash\":true,\"DumpEmptyDatabase\":true,\"PosAfterConnect\":false,\"CompressType\":0,\"Host\":\"127.0.0.1\",\"Port\":4000,\"Threads\":4,\"User\":\"root\",\"Security\":{\"CAPath\":\"\",\"CertPath\":\"\",\"KeyPath\":\"\"},\"LogLevel\":\"info\",\"LogFile\":\"dumpling.log\",\"LogFormat\":\"text\",\"OutputDirPath\":\"./export-2021-04-13T16:31:56+08:00\",\"StatusAddr\":\":8281\",\"Snapshot\":\"424228347403829254\",\"Consistency\":\"snapshot\",\"CsvNullValue\":\"\\\\N\",\"SQL\":\"\",\"CsvSeparator\":\",\",\"CsvDelimiter\":\"\\\"\",\"Databases\":[],\"Where\":\"\",\"FileType\":\"sql\",\"ServerInfo\":{\"ServerType\":3,\"ServerVersion\":\"4.0.11\"},\"Rows\":0,\"ReadTimeout\":900000000000,\"TiDBMemQuotaQuery\":0,\"FileSize\":0,\"StatementSize\":1000000,\"SessionParams\":{\"tidb_snapshot\":\"424228347403829254\"},\"Tables\":null}"]
[2021/04/13 16:31:57.093 +08:00] [INFO] [collector.go:212] ["backup Success summary: total backup ranges: 1, total success: 1, total failed: 0, total take(backup time): 1.207569ms, total take(real time): 1.264326ms"]
[2021/04/13 16:31:57.093 +08:00] [INFO] [main.go:81] ["dump data successfully, dumpling will exit now"]
```

Related changes

 - Need to cherry-pick to the release branch
 
### Release note

<!-- bugfixes or new feature need a release note, must in the form of a list, such as

- support -T/--tables-list argument

or if no need to be included in the release note, just add the following line

- No release note
-->
- No release note


ti-chi-bot: [REVIEW NOTIFICATION]

This pull request has not been approved.


To complete the [pull request process](https://book.prow.tidb.io/#/en/workflows/pr), please ask the reviewers in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/270/owners) to review by filling `/cc @reviewer` in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/270/owners) by filling  `/assign @committer` in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found [here](https://prow.tidb.io/command-help?repo=pingcap%2Fdumpling).

<details>

Reviewer can indicate their review by writing `/lgtm` in a comment.
Reviewer can cancel approval by writing `/lgtm cancel` in a comment.
</details>

<!--Review Notification Identifier-->
ti-chi-bot: [REVIEW NOTIFICATION]

This pull request has been approved by:

- lance6716




To complete the [pull request process](https://book.prow.tidb.io/#/en/workflows/pr), please ask the reviewers in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/270/owners) to review by filling `/cc @reviewer` in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the [list](https://prow.tidb.io/tichi/repos/pingcap/dumpling/pulls/270/owners) by filling  `/assign @committer` in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found [here](https://prow.tidb.io/command-help?repo=pingcap%2Fdumpling).

<details>

Reviewer can indicate their review by writing `/lgtm` in a comment.
Reviewer can cancel approval by writing `/lgtm cancel` in a comment.
</details>

<!--Review Notification Identifier-->
lichunzhu: /merge
ti-chi-bot: This pull request has been accepted and is ready to merge. <details>Commit hash: 810cbdcaf6c66a57468c9d64d779a213022a635a</details>

Patch:
diff --git a/v4/export/dump.go b/v4/export/dump.go
index 1e8f613..e3d7274 100755
--- a/v4/export/dump.go
+++ b/v4/export/dump.go
@@ -21,6 +21,7 @@ import (
 	"github.com/pingcap/br/pkg/summary"
 	"github.com/pingcap/errors"
 	"github.com/pingcap/failpoint"
+	pclog "github.com/pingcap/log"
 	pd "github.com/tikv/pd/client"
 	"go.uber.org/zap"
 	"golang.org/x/sync/errgroup"
@@ -766,12 +767,16 @@ func runSteps(d *Dumper, steps ...func(*Dumper) error) error {
 
 func initLogger(d *Dumper) error {
 	conf := d.conf
-	var logger log.Logger
+	var (
+		logger log.Logger
+		err    error
+		props  *pclog.ZapProperties
+	)
+	// conf.Logger != nil means dumpling is used as a library
 	if conf.Logger != nil {
 		logger = log.NewAppLogger(conf.Logger)
 	} else {
-		var err error
-		logger, err = log.InitAppLogger(&log.Config{
+		logger, props, err = log.InitAppLogger(&log.Config{
 			Level:  conf.LogLevel,
 			File:   conf.LogFile,
 			Format: conf.LogFormat,
@@ -779,6 +784,7 @@ func initLogger(d *Dumper) error {
 		if err != nil {
 			return errors.Trace(err)
 		}
+		pclog.ReplaceGlobals(logger.Logger, props)
 		cli.LogLongVersion(logger)
 	}
 	d.tctx = d.tctx.WithLogger(logger)
diff --git a/v4/log/log.go b/v4/log/log.go
index 4632634..79f840d 100644
--- a/v4/log/log.go
+++ b/v4/log/log.go
@@ -38,8 +38,8 @@ type Config struct {
 }
 
 // InitAppLogger inits the wrapped logger from config.
-func InitAppLogger(cfg *Config) (Logger, error) {
-	logger, _, err := pclog.InitLogger(&pclog.Config{
+func InitAppLogger(cfg *Config) (Logger, *pclog.ZapProperties, error) {
+	logger, props, err := pclog.InitLogger(&pclog.Config{
 		Level: cfg.Level,
 		File: pclog.FileLogConfig{
 			Filename:   cfg.File,
@@ -50,9 +50,9 @@ func InitAppLogger(cfg *Config) (Logger, error) {
 		Format: cfg.Format,
 	})
 	if err != nil {
-		return appLogger, errors.Trace(err)
+		return appLogger, props, errors.Trace(err)
 	}
-	return Logger{logger.WithOptions(zap.AddStacktrace(zap.DPanicLevel))}, nil
+	return Logger{logger.WithOptions(zap.AddStacktrace(zap.DPanicLevel))}, props, nil
 }
 
 // NewAppLogger returns the wrapped logger from config.


Test patch:
diff --git a/v4/export/writer_util_test.go b/v4/export/writer_util_test.go
index cac38c9..ff249c5 100644
--- a/v4/export/writer_util_test.go
+++ b/v4/export/writer_util_test.go
@@ -20,7 +20,7 @@ var appLogger log.Logger
 
 func TestT(t *testing.T) {
 	initColTypeRowReceiverMap()
-	logger, err := log.InitAppLogger(&log.Config{
+	logger, _, err := log.InitAppLogger(&log.Config{
 		Level:  "info",
 		File:   "",
 		Format: "text",