Repository: pingcap/tidb (Go)

Issue:
Got `Query execution was interrupted` when graceful shutdown
## Description
## Bug Report

Please answer these questions before submitting your issue. Thanks!

### 1. Minimal reproduce step (Required)

<!-- a step by step guide for reproducing the bug. -->

```mysql
create table t(a int);
select 1 from t where (select sleep(3)) ;
```

During the sleeping, kill(sig terminate) the tidb-server

### 2. What did you expect to see? (Required)

```
Empty set (3.01 sec)
```
And after the result returned, the server exits gracefully.

### 3. What did you see instead (Required)

```
ERROR 1317 (70100): Query execution was interrupted
```
### 4. What is your TiDB version? (Required)

<!-- Paste the output of SELECT tidb_version() -->





## SIG slack channel
[#sig-exec](https://slack.tidb.io/invite?team=tidb-community&channel=sig-exec&ref=high-performance)

## Score
- 300

## Mentor
 * @qw4990



Discussion:
@tiancaiamao please take a look.
I think the current behavior is not a bug.
When kill or close TiDB, the integrity of the data may not be guaranteed, so return error is a better choice.
 @zz-jason @SunRunAway 
I can't disagree more. TiDB should do its best efforts for serving the current queries when graceful closing, unless it reaches the graceful timeout. 
What's the behavior in MySQL?
> the integrity of the data may not be guaranteed

I think this is not an issue. When a server facing `kill -9`, the client may also receive the half data and it should treat `connection reset by peer` correctly.
This issue does not belong to any SIG.

It has been added to the challenge program, but your issue description has some problems.

<details>
<summary>More</summary>

**Tip**    : 
    You need to ensure that the description of the issue follows the following template:
    
    ```
    ## Score

    - ${score}

    ## Mentor

    - ${mentor}
    ```
    

**Warning**: The description format for this issue is wrong.
</details>
    
The issue has been removed from the challenge program.
> When a server facing kill -9, the client may also receive the half data and it should treat connection reset by peer correctly.

That's totally a different comparison.
If the peer socket is reset, the error is from the network layer and the OS would notify the process about the error. And of course, the client knows how to handle the error.
If the data itself is wrong, and if tidb return the wrong data with OK response message, the client thinks every thing goes well! For example, the database should return 100 lines but just returns 50 lines, is it OK?

It's ALL ABOUT CORRECTNESS
What situation would tidb return half-truncated data with an OK response?
> What situation would tidb return half-truncated data with an OK response?

Interrupted, or graceful shutdown here.
The code may be 

```
Next() ... 
Next() ... 
detect killed signal ...  
(we can't return success to client here)
Next() not called ...
```
From confirmation, I implemented a simple SQL client to verify what receives when meets a kill, see https://gist.github.com/SunRunAway/7dcb3fce0ceb448e597163ca9ef00a4c

1. run a server based on my branch with a debug sleeping, see https://github.com/pingcap/tidb/compare/master...SunRunAway:isseue19663-test?expand=1#diff-dc417f5322100e2e83a8818c70a25878R1633-R1635
2. create a table with 16384 rows
3. run a simple client `go run sql.go -P 4000 -D test 'select * from t'` with my client. 
4. kill the connection or query with another mysql client.
5. the client from step 3 says
```
4000
select * from t
2020/09/07 22:22:00 sql.go:54: 1000
2020/09/07 22:22:00 sql.go:54: 2000
2020/09/07 22:22:00 sql.go:54: 3000
2020/09/07 22:22:00 sql.go:59: Error 1317: Query execution was interrupted
exit status 1
```

So If the client handles the error correctly, the correctness is ensured.


---
Besides, this situation exists before my PR. If a server is running [`writeChunks`](https://github.com/pingcap/tidb/blob/e3a93099729a91068a1e969e264138e08be1b668/server/conn.go#L1606) when receives a kill, next `rs.Next` will still return an error after the first EOF with schema sent.
## Please edit this comment or add a new comment to complete the following information

### Not a bug

1. Remove the 'type/bug' label
2. Add notes to indicate why it is not a bug

### Duplicate bug

1. Add the 'type/duplicate' label
2. Add the link to the original bug

### Bug

Note: Make Sure that 'component', and 'severity' labels are added
Example for how to fill out the template: https://github.com/pingcap/tidb/issues/20100

#### 1. Root Cause Analysis (RCA) (optional) 
<!-- Write down the reason why this bug occurs -->

#### 2. Symptom (optional)

<!-- What will the user see when this bug occurs. The error message may be in the terminal, log or monitoring -->

#### 3. All Trigger Conditions (optional)

<!-- All the user scenarios that may trigger this bug -->

#### 4. Workaround (optional)

#### 5. Affected versions

<!--
In the format of [start_version:end_version], multiple version ranges are
accepted. If the bug only affects the unreleased version, please input:
"unreleased". For example:

Notes:
  1. Do not use any white spaces in '[]'.
  2. The range in '[]' is a closed interval
  3. The version format is `v$Major.$Minor.$Patch`, the $Majoy and $Minor
     number in a version range should be the same. [v3.0.1:v3.1.2] is
     invalid because the $Minor number of the version range is different.

Example 1: [v3.0.1:v3.0.5], [v4.0.1:v4.0.5]
Example 2: unreleased
-->

#### 6. Fixed versions

<!--
The first released version that contains this fix in each minor version. If the bug's affected version has been released, the fixed version should be a detailed version number; If the bug doesn't affect any released version, the fixed version can be "master";  

Example 1: v3.0.13, v4.0.5
Example 2: master
-->
@SunRunAway  You can't say, "I test a case, it's correct, so it's correct in all cases."  
> @SunRunAway You can't say, "I test a case, it's correct, so it's correct in all cases."

What do you mean of it?



Patch:
diff --git a/server/conn.go b/server/conn.go
index 081fe4aea81fe..918e50563fcc7 100644
--- a/server/conn.go
+++ b/server/conn.go
@@ -750,7 +750,10 @@ func (cc *clientConn) Run(ctx context.Context) {
 	// The client connection would detect the events when it fails to change status
 	// by CAS operation, it would then take some actions accordingly.
 	for {
-		if !atomic.CompareAndSwapInt32(&cc.status, connStatusDispatching, connStatusReading) {
+		if !atomic.CompareAndSwapInt32(&cc.status, connStatusDispatching, connStatusReading) ||
+			// The judge below will not be hit by all means,
+			// But keep it stayed as a reminder and for the code reference for connStatusWaitShutdown.
+			atomic.LoadInt32(&cc.status) == connStatusWaitShutdown {
 			return
 		}
 
@@ -829,8 +832,8 @@ func (cc *clientConn) ShutdownOrNotify() bool {
 		return true
 	}
 	// If the client connection status is dispatching, we can't shutdown it immediately,
-	// so set the status to WaitShutdown as a notification, the client will detect it
-	// and then exit.
+	// so set the status to WaitShutdown as a notification, the loop in clientConn.Run
+	// will detect it and then exit.
 	atomic.StoreInt32(&cc.status, connStatusWaitShutdown)
 	return false
 }
@@ -1575,7 +1578,7 @@ func (cc *clientConn) handleStmt(ctx context.Context, stmt ast.StmtNode, warns [
 
 	if rs != nil {
 		connStatus := atomic.LoadInt32(&cc.status)
-		if connStatus == connStatusShutdown || connStatus == connStatusWaitShutdown {
+		if connStatus == connStatusShutdown {
 			return executor.ErrQueryInterrupted
 		}
 
diff --git a/server/server.go b/server/server.go
index edb290bf9ad74..345948ac2b452 100644
--- a/server/server.go
+++ b/server/server.go
@@ -524,9 +524,6 @@ func (s *Server) ShowProcessList() map[uint64]*util.ProcessInfo {
 	defer s.rwlock.RUnlock()
 	rs := make(map[uint64]*util.ProcessInfo, len(s.clients))
 	for _, client := range s.clients {
-		if atomic.LoadInt32(&client.status) == connStatusWaitShutdown {
-			continue
-		}
 		if pi := client.ctx.ShowProcess(); pi != nil {
 			rs[pi.ID] = pi
 		}
@@ -539,7 +536,7 @@ func (s *Server) GetProcessInfo(id uint64) (*util.ProcessInfo, bool) {
 	s.rwlock.RLock()
 	conn, ok := s.clients[id]
 	s.rwlock.RUnlock()
-	if !ok || atomic.LoadInt32(&conn.status) == connStatusWaitShutdown {
+	if !ok {
 		return &util.ProcessInfo{}, false
 	}
 	return conn.ctx.ShowProcess(), ok
@@ -558,7 +555,7 @@ func (s *Server) Kill(connectionID uint64, query bool) {
 	}
 
 	if !query {
-		// Mark the client connection status as WaitShutdown, when the goroutine detect
+		// Mark the client connection status as WaitShutdown, when clientConn.Run detect
 		// this, it will end the dispatch loop and exit.
 		atomic.StoreInt32(&conn.status, connStatusWaitShutdown)
 	}


Test patch:
diff --git a/tests/graceshutdown/.gitignore b/tests/graceshutdown/.gitignore
new file mode 100644
index 0000000000000..3521e226f227a
--- /dev/null
+++ b/tests/graceshutdown/.gitignore
@@ -0,0 +1,1 @@
+tidb-slow.log
diff --git a/tests/graceshutdown/Makefile b/tests/graceshutdown/Makefile
new file mode 100644
index 0000000000000..f1f028514b1c0
--- /dev/null
+++ b/tests/graceshutdown/Makefile
@@ -0,0 +1,33 @@
+# Copyright 2020 PingCAP, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+BASE_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/../..)
+OUT_DIR := $(abspath $(dir $(lastword $(MAKEFILE_LIST)))/bin)
+
+include $(BASE_DIR)/Makefile.common
+
+.PHONY: server buildsucc
+
+default: server buildsucc
+
+buildsucc:
+	@echo "Build TiDB Server successfully!"
+
+server:
+ifeq ($(TARGET), "")
+	cd ${BASE_DIR} && \
+	CGO_ENABLED=1 $(GOBUILD) -ldflags '$(LDFLAGS)' -o $(OUT_DIR)/tidb-server tidb-server/main.go
+else
+	cd ${BASE_DIR} && \
+	CGO_ENABLED=1 $(GOBUILD) -ldflags '$(LDFLAGS)' -o '$(TARGET)' tidb-server/main.go
+endif
diff --git a/tests/graceshutdown/go.mod b/tests/graceshutdown/go.mod
new file mode 100644
index 0000000000000..0b1063caec2ad
--- /dev/null
+++ b/tests/graceshutdown/go.mod
@@ -0,0 +1,10 @@
+module graceshutdown
+
+go 1.15
+
+require (
+	github.com/go-sql-driver/mysql v1.5.0
+	github.com/juju/errors v0.0.0-20200330140219-3fe23663418f
+	github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712
+	github.com/sirupsen/logrus v1.7.0
+)
diff --git a/tests/graceshutdown/go.sum b/tests/graceshutdown/go.sum
new file mode 100644
index 0000000000000..086b125d80d92
--- /dev/null
+++ b/tests/graceshutdown/go.sum
@@ -0,0 +1,64 @@
+github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
+github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/go-sql-driver/mysql v1.5.0 h1:ozyZYNQW3x3HtqT1jira07DN2PArx2v7/mN66gGcHOs=
+github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
+github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
+github.com/juju/errors v0.0.0-20200330140219-3fe23663418f h1:MCOvExGLpaSIzLYB4iQXEHP4jYVU6vmzLNQPdMVrxnM=
+github.com/juju/errors v0.0.0-20200330140219-3fe23663418f/go.mod h1:W54LbzXuIE0boCoNJfwqpmkKJ1O4TCTZMetAt6jGk7Q=
+github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
+github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
+github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
+github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/pingcap/check v0.0.0-20190102082844-67f458068fc8/go.mod h1:B1+S9LNcuMyLH/4HMTViQOJevkGiik3wW2AN9zb2fNQ=
+github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712 h1:R8gStypOBmpnHEx1qi//SaqxJVI4inOqljg/Aj5/390=
+github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712/go.mod h1:PYMCGwN0JHjoqGr3HrZoD+b8Tgx8bKnArhSq8YVzUMc=
+github.com/pingcap/errors v0.11.0/go.mod h1:Oi8TUi2kEtXXLMJk9l1cGmz20kV3TaQ0usTwv5KuLY8=
+github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9 h1:AJD9pZYm72vMgPcQDww9rkZ1DnWfl0pXV3BOWlkYIjA=
+github.com/pingcap/log v0.0.0-20191012051959-b742a5d432e9/go.mod h1:4rbK1p9ILyIfb6hU7OG2CiWSqMXnp3JMbiaVJ6mvoY8=
+github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/rogpeppe/go-internal v1.3.0/go.mod h1:M8bDsm7K2OlrFYOpmOWEs/qY81heoFRclV5y23lUDJ4=
+github.com/sirupsen/logrus v1.7.0 h1:ShrD1U9pZB12TX0cVy0DtePoCH97K8EtX+mg7ZARUtM=
+github.com/sirupsen/logrus v1.7.0/go.mod h1:yWOB1SBYBC5VeMP7gHvWumXLIWorT60ONWic61uBYv0=
+github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
+github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
+github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4=
+go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
+go.uber.org/atomic v1.5.0 h1:OI5t8sDa1Or+q8AeE+yKeB/SDYioSHAgcVljj9JIETY=
+go.uber.org/atomic v1.5.0/go.mod h1:sABNBOSYdrvTF6hTgEIbc7YasKWGhgEQZyfxyTvoXHQ=
+go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
+go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
+go.uber.org/multierr v1.4.0 h1:f3WCSC2KzAcBXGATIxAB1E2XuCpNU255wNKZ505qi3E=
+go.uber.org/multierr v1.4.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4=
+go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA=
+go.uber.org/zap v1.9.1/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
+go.uber.org/zap v1.12.0 h1:dySoUQPFBGj6xwjmBzageVL8jGi8uxc6bEmJQjA06bw=
+go.uber.org/zap v1.12.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM=
+golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
+golang.org/x/crypto v0.0.0-20190510104115-cbcb75029529/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mod v0.0.0-20190513183733-4bf6d317e70e/go.mod h1:mXi4GBBbnImb6dmsKGUJ2LatrhH/nqhxcFungHvyanc=
+golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
+golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
+golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037 h1:YyJpGZS1sBuBCzLAR1VEpK193GlqGZbnPFnPV/5Rsb4=
+golang.org/x/sys v0.0.0-20191026070338-33540a1f6037/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
+golang.org/x/tools v0.0.0-20190621195816-6e04913cbbac/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc=
+golang.org/x/tools v0.0.0-20191029041327-9cc4af7d6b2c/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/tools v0.0.0-20191107010934-f79515f33823/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
+golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/errgo.v2 v2.1.0/go.mod h1:hNsd1EY+bozCKY1Ytp96fpM3vjJbqLJn88ws8XvfDNI=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0 h1:1Lc07Kr7qY4U2YPouBjpCLxpiyxIVoxqXgkXLknAOE8=
+gopkg.in/natefinch/lumberjack.v2 v2.0.0/go.mod h1:l0ndWWf7gzL7RNwBG7wST/UCcT4T24xpD6X8LsfU/+k=
+gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
+honnef.co/go/tools v0.0.1-2019.2.3/go.mod h1:a3bituU0lyd329TUQxRnasdCoJDkEUEAqEt0JzvZhAg=
diff --git a/tests/graceshutdown/graceshutdown_test.go b/tests/graceshutdown/graceshutdown_test.go
new file mode 100644
index 0000000000000..c90bd82d028d6
--- /dev/null
+++ b/tests/graceshutdown/graceshutdown_test.go
@@ -0,0 +1,146 @@
+// Copyright 2020 PingCAP, Inc.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package graceshutdown
+
+import (
+	"context"
+	"database/sql"
+	"flag"
+	"fmt"
+	"os"
+	"os/exec"
+	"testing"
+	"time"
+
+	_ "github.com/go-sql-driver/mysql"
+	"github.com/juju/errors"
+	. "github.com/pingcap/check"
+	log "github.com/sirupsen/logrus"
+)
+
+var (
+	tidbBinaryPath = flag.String("s", "bin/tidb-server", "tidb server binary path")
+	tmpPath        = flag.String("tmp", "/tmp/tidb_gracefulshutdown", "temporary files path")
+	tidbStartPort  = flag.Int("tidb_start_port", 5500, "first tidb server listening port")
+	tidbStatusPort = flag.Int("tidb_status_port", 8500, "first tidb server status port")
+)
+
+func TestGracefulShutdown(t *testing.T) {
+	CustomVerboseFlag = true
+	TestingT(t)
+}
+
+var _ = Suite(&TestGracefulShutdownSuite{})
+
+type TestGracefulShutdownSuite struct {
+}
+
+func (s *TestGracefulShutdownSuite) SetUpSuite(c *C) {
+}
+func (s *TestGracefulShutdownSuite) TearDownSuite(c *C) {
+}
+
+func (s *TestGracefulShutdownSuite) startTiDBWithoutPD(port int, statusPort int) (cmd *exec.Cmd, err error) {
+	cmd = exec.Command(*tidbBinaryPath,
+		"--store=mocktikv",
+		fmt.Sprintf("--path=%s/mocktikv", *tmpPath),
+		fmt.Sprintf("-P=%d", port),
+		fmt.Sprintf("--status=%d", statusPort),
+		fmt.Sprintf("--log-file=%s/tidb%d.log", *tmpPath, port))
+	log.Infof("starting tidb: %v", cmd)
+	err = cmd.Start()
+	if err != nil {
+		return nil, errors.Trace(err)
+	}
+	time.Sleep(500 * time.Millisecond)
+	return cmd, nil
+}
+
+func (s *TestGracefulShutdownSuite) stopService(name string, cmd *exec.Cmd) (err error) {
+	if err = cmd.Process.Signal(os.Interrupt); err != nil {
+		return errors.Trace(err)
+	}
+	log.Infof("service \"%s\" Interrupt", name)
+	if err = cmd.Wait(); err != nil {
+		return errors.Trace(err)
+	}
+	log.Infof("service \"%s\" stopped gracefully", name)
+	return nil
+}
+
+func (s *TestGracefulShutdownSuite) connectTiDB(port int) (db *sql.DB, err error) {
+	addr := fmt.Sprintf("127.0.0.1:%d", port)
+	dsn := fmt.Sprintf("root@(%s)/test", addr)
+	sleepTime := 250 * time.Millisecond
+	startTime := time.Now()
+	for i := 0; i < 5; i++ {
+		db, err = sql.Open("mysql", dsn)
+		if err != nil {
+			log.Warnf("open addr %v failed, retry count %d err %v", addr, i, err)
+			continue
+		}
+		err = db.Ping()
+		if err == nil {
+			break
+		}
+		log.Warnf("ping addr %v failed, retry count %d err %v", addr, i, err)
+
+		db.Close()
+		time.Sleep(sleepTime)
+		sleepTime += sleepTime
+	}
+	if err != nil {
+		log.Errorf("connect to server addr %v failed %v, take time %v", addr, err, time.Since(startTime))
+		return nil, errors.Trace(err)
+	}
+	db.SetMaxOpenConns(10)
+
+	log.Infof("connect to server %s ok", addr)
+	return db, nil
+}
+
+func (s *TestGracefulShutdownSuite) TestGracefulShutdown(c *C) {
+	port := *tidbStartPort + 1
+	tidb, err := s.startTiDBWithoutPD(port, *tidbStatusPort)
+	c.Assert(err, IsNil)
+
+	db, err := s.connectTiDB(port)
+	c.Assert(err, IsNil)
+	defer db.Close()
+
+	ctx, cancel := context.WithDeadline(context.Background(), time.Now().Add(10*time.Second))
+	defer cancel()
+	conn1, err := db.Conn(ctx)
+	c.Assert(err, IsNil)
+	defer conn1.Close()
+
+	_, err = conn1.ExecContext(ctx, "drop table if exists t;")
+	c.Assert(err, IsNil)
+	_, err = conn1.ExecContext(ctx, "create table t(a int);")
+	c.Assert(err, IsNil)
+	_, err = conn1.ExecContext(ctx, "insert into t values(1);")
+	c.Assert(err, IsNil)
+
+	go func() {
+		time.Sleep(1e9)
+		err = s.stopService("tidb", tidb)
+		c.Assert(err, IsNil)
+	}()
+
+	sql := `select 1 from t where not (select sleep(3)) ;`
+	var a int64
+	err = conn1.QueryRowContext(ctx, sql).Scan(&a)
+	c.Assert(err, IsNil)
+	c.Assert(a, Equals, int64(1))
+}
diff --git a/tests/graceshutdown/run-tests.sh b/tests/graceshutdown/run-tests.sh
new file mode 100755
index 0000000000000..a4def89eaf845
--- /dev/null
+++ b/tests/graceshutdown/run-tests.sh
@@ -0,0 +1,18 @@
+#!/usr/bin/env bash
+# Copyright 2020 PingCAP, Inc.
+
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+
+#     http://www.apache.org/licenses/LICENSE-2.0
+
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eu
+trap 'set +e; PIDS=$(jobs -p); [ -n "$PIDS" ] && kill -9 $PIDS' EXIT
+
+go test