Repository: housepower/ClickHouse-Native-JDBC
Pull request #401 (2022-01-25 09:20:30)

#400 ClickhouseResultSet add readRows and readBytes
Like `clickhouse-client` return readRows and readBytes.

ReadRow's value equals column`read_rows` of `system.query_log`
ReadBytes value equals column`read_bytes` of `system.query_log`

if `ResultSet` has next value, readRows or readBytes will not exactly the same


Usage
```
((ClickHouseResultSet) rs).getReadRows()
((ClickHouseResultSet) rs).getReadBytes()
```



codecov[bot]: # [Codecov](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) Report
> Merging [#401](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (d1ed7ea) into [master](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/commit/bb097619c220fd24a7c0f25b27241a85c21563e5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) (bb09761) will **increase** coverage by `0.20%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401/graphs/tree.svg?width=650&height=150&src=pr&token=Izz9Qy1I72&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)

```diff
@@             Coverage Diff              @@
##             master     #401      +/-   ##
============================================
+ Coverage     62.40%   62.60%   +0.20%     
- Complexity     1246     1256      +10     
============================================
  Files           135      135              
  Lines          6633     6662      +29     
  Branches        514      519       +5     
============================================
+ Hits           4139     4171      +32     
+ Misses         2219     2216       -3     
  Partials        275      275              
```


| [Impacted Files](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None) | Coverage Δ | |
|---|---|---|
| [...rc/main/java/com/github/housepower/data/Block.java](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-Y2xpY2tob3VzZS1uYXRpdmUtamRiYy9zcmMvbWFpbi9qYXZhL2NvbS9naXRodWIvaG91c2Vwb3dlci9kYXRhL0Jsb2NrLmphdmE=) | `90.66% <100.00%> (+0.96%)` | :arrow_up: |
| [...om/github/housepower/jdbc/ClickHouseResultSet.java](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-Y2xpY2tob3VzZS1uYXRpdmUtamRiYy9zcmMvbWFpbi9qYXZhL2NvbS9naXRodWIvaG91c2Vwb3dlci9qZGJjL0NsaWNrSG91c2VSZXN1bHRTZXQuamF2YQ==) | `64.07% <100.00%> (+4.20%)` | :arrow_up: |
| [...ithub/housepower/stream/ClickHouseQueryResult.java](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-Y2xpY2tob3VzZS1uYXRpdmUtamRiYy9zcmMvbWFpbi9qYXZhL2NvbS9naXRodWIvaG91c2Vwb3dlci9zdHJlYW0vQ2xpY2tIb3VzZVF1ZXJ5UmVzdWx0LmphdmE=) | `88.23% <100.00%> (+3.05%)` | :arrow_up: |
| [...m/github/housepower/protocol/ProgressResponse.java](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None#diff-Y2xpY2tob3VzZS1uYXRpdmUtamRiYy9zcmMvbWFpbi9qYXZhL2NvbS9naXRodWIvaG91c2Vwb3dlci9wcm90b2NvbC9Qcm9ncmVzc1Jlc3BvbnNlLmphdmE=) | `90.00% <0.00%> (+20.00%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Last update [bb09761...d1ed7ea](https://codecov.io/gh/housepower/ClickHouse-Native-JDBC/pull/401?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=None).


Patch:
diff --git a/clickhouse-native-jdbc/src/main/java/com/github/housepower/data/Block.java b/clickhouse-native-jdbc/src/main/java/com/github/housepower/data/Block.java
index 1c7a756..7825af5 100644
--- a/clickhouse-native-jdbc/src/main/java/com/github/housepower/data/Block.java
+++ b/clickhouse-native-jdbc/src/main/java/com/github/housepower/data/Block.java
@@ -55,6 +55,8 @@ public class Block {
     private final Object[] rowData;
     private final int[] placeholderIndexes;
     private int rowCnt;
+    private long readRows = 0;
+    private long readBytes = 0;
 
     public Block() {
         this(0, new IColumn[0]);
@@ -169,4 +171,18 @@ public class Block {
             }
         }
     }
+
+    public void setProgress(long readRows, long readBytes) {
+        this.readRows = readRows;
+        this.readBytes = readBytes;
+    }
+
+    public long readRows() {
+        return readRows;
+    }
+
+    public long readBytes() {
+        return readBytes;
+    }
+
 }
diff --git a/clickhouse-native-jdbc/src/main/java/com/github/housepower/jdbc/ClickHouseResultSet.java b/clickhouse-native-jdbc/src/main/java/com/github/housepower/jdbc/ClickHouseResultSet.java
index 0d3a2a8..ba957f5 100644
--- a/clickhouse-native-jdbc/src/main/java/com/github/housepower/jdbc/ClickHouseResultSet.java
+++ b/clickhouse-native-jdbc/src/main/java/com/github/housepower/jdbc/ClickHouseResultSet.java
@@ -56,6 +56,8 @@ public class ClickHouseResultSet implements SQLResultSet {
     private boolean isFirst = false;
     private boolean isAfterLast = false;
     private boolean isClosed = false;
+    private long readRows = 0;
+    private long readBytes = 0;
 
     public ClickHouseResultSet(ClickHouseStatement statement,
                                ClickHouseConfig cfg,
@@ -406,7 +408,13 @@ public class ClickHouseResultSet implements SQLResultSet {
         boolean isBeforeFirst = isBeforeFirst();
         LOG.trace("check status[before]: is_before_first: {}, is_first: {}, is_after_last: {}", isBeforeFirst, isFirst, isAfterLast);
 
-        boolean hasNext = ++currentRowNum < currentBlock.rowCnt() || (currentRowNum = 0) < (currentBlock = fetchBlock()).rowCnt();
+        boolean hasNext = ++currentRowNum < currentBlock.rowCnt();
+
+        if (!hasNext) {
+            hasNext = (currentRowNum = 0) < (currentBlock = fetchBlock()).rowCnt();
+            readRows += currentBlock.readRows();
+            readBytes += currentBlock.readBytes();
+        }
 
         isFirst = isBeforeFirst && hasNext;
         isAfterLast = !hasNext;
@@ -435,14 +443,33 @@ public class ClickHouseResultSet implements SQLResultSet {
     }
 
     private Block fetchBlock() throws SQLException {
+        long readRows = 0;
+        long readBytes = 0;
+
         while (dataResponses.hasNext()) {
             LOG.trace("fetch next DataResponse");
             DataResponse next = dataResponses.next();
+
+            readRows += next.block().readRows();
+            readBytes += next.block().readBytes();
+
             if (next.block().rowCnt() > 0) {
+                next.block().setProgress(readRows, readBytes);
                 return next.block();
             }
         }
         LOG.debug("no more DataResponse, return empty Block");
-        return new Block();
+
+        Block bk = new Block();
+        bk.setProgress(readRows, readBytes);
+        return bk;
+    }
+
+    public long getReadRows() {
+        return this.readRows;
+    }
+
+    public long getReadBytes() {
+        return this.readBytes;
     }
 }
diff --git a/clickhouse-native-jdbc/src/main/java/com/github/housepower/stream/ClickHouseQueryResult.java b/clickhouse-native-jdbc/src/main/java/com/github/housepower/stream/ClickHouseQueryResult.java
index 25d3821..44b96c6 100644
--- a/clickhouse-native-jdbc/src/main/java/com/github/housepower/stream/ClickHouseQueryResult.java
+++ b/clickhouse-native-jdbc/src/main/java/com/github/housepower/stream/ClickHouseQueryResult.java
@@ -19,6 +19,7 @@ import com.github.housepower.misc.CheckedIterator;
 import com.github.housepower.misc.CheckedSupplier;
 import com.github.housepower.protocol.DataResponse;
 import com.github.housepower.protocol.EOFStreamResponse;
+import com.github.housepower.protocol.ProgressResponse;
 import com.github.housepower.protocol.Response;
 
 import java.sql.SQLException;
@@ -85,12 +86,19 @@ public class ClickHouseQueryResult implements QueryResult {
     }
 
     private DataResponse consumeDataResponse() throws SQLException {
+        long readRows = 0;
+        long readBytes = 0;
         while (!atEnd) {
             Response response = responseSupplier.get();
             if (response instanceof DataResponse) {
-                return (DataResponse) response;
+                DataResponse dataResponse = (DataResponse) response;
+                dataResponse.block().setProgress(readRows, readBytes);
+                return dataResponse;
             } else if (response instanceof EOFStreamResponse || response == null) {
                 atEnd = true;
+            } else if (response instanceof ProgressResponse) {
+                readRows += ((ProgressResponse) response).newRows();
+                readBytes += ((ProgressResponse) response).newBytes();
             }
         }
 


Test patch:
diff --git a/clickhouse-native-jdbc/src/test/java/com/github/housepower/jdbc/QueryScanStatTest.java b/clickhouse-native-jdbc/src/test/java/com/github/housepower/jdbc/QueryScanStatTest.java
new file mode 100644
index 0000000..491b4ac
--- /dev/null
+++ b/clickhouse-native-jdbc/src/test/java/com/github/housepower/jdbc/QueryScanStatTest.java
@@ -0,0 +1,105 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.github.housepower.jdbc;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.BeforeEach;
+
+import java.sql.Connection;
+import java.sql.DriverManager;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.util.Locale;
+import java.util.UUID;
+
+import static org.junit.Assert.assertEquals;
+
+public class QueryScanStatTest extends AbstractITest {
+
+    @BeforeEach
+    public void init() throws SQLException {
+        resetDriverManager();
+    }
+
+    @Test
+    public void readRowsAndReadBytes() throws Exception {
+        String queryId = UUID.randomUUID().toString();
+        double random = Math.random();
+        long lines = (long) (random * 100000000);
+
+        try (Connection connection = DriverManager
+                .getConnection(String.format(Locale.ROOT, "jdbc:clickhouse://%s:%s?query_id=%s", CK_HOST, CK_PORT,
+                        queryId))
+        ) {
+            withStatement(connection, stmt -> {
+                stmt.executeQuery("DROP TABLE IF EXISTS test_scan_stat1");
+                stmt.executeQuery("CREATE TABLE test_scan_stat1 "
+                        + "(c1 UInt32) "
+                        + "ENGINE = MergeTree() "
+                        + "PARTITION BY tuple()"
+                        + "ORDER BY tuple() ");
+                stmt.executeQuery("INSERT INTO test_scan_stat1 SELECT number FROM system.numbers LIMIT " + lines);
+
+                long readRows;
+                long readBytes;
+
+                try (ResultSet rs = stmt.executeQuery("SELECT c1 FROM test_scan_stat1 LIMIT 100")) {
+                    while (rs.next()) {
+                        // ignore result
+                    }
+
+                    readRows = ((ClickHouseResultSet) rs).getReadRows();
+                    readBytes = ((ClickHouseResultSet) rs).getReadBytes();
+                }
+
+                Thread.sleep(12 * 1000); // wait insert into ch log table
+
+                try (ResultSet rs = stmt.executeQuery(String.format(Locale.ROOT,
+                        "SELECT read_rows,read_bytes FROM system.query_log WHERE query_id='%s'"
+                                + " and type='QueryFinish' and startsWith(query, 'SELECT') ORDER BY query_start_time asc", queryId))) {
+                    if (rs.next()) {
+                        assertEquals(rs.getLong("read_rows"), readRows);
+                        assertEquals(rs.getLong("read_bytes"), readBytes);
+                    }
+                }
+            });
+        }
+    }
+
+    @Test
+    public void readRows() throws Exception {
+        double random = Math.random();
+        long lines = (long) (random * 100000000);
+
+
+        withStatement( stmt -> {
+            stmt.executeQuery("DROP TABLE IF EXISTS test_scan_stat2");
+            stmt.executeQuery("CREATE TABLE test_scan_stat2 "
+                    + "(c1 UInt32) "
+                    + "ENGINE = MergeTree() "
+                    + "PARTITION BY tuple()"
+                    + "ORDER BY tuple() ");
+            stmt.executeQuery("INSERT INTO test_scan_stat2 SELECT number FROM system.numbers LIMIT " + lines);
+
+            try (ResultSet rs = stmt.executeQuery("SELECT count(distinct c1) FROM test_scan_stat2")) {
+                while (rs.next()) {
+                    // ignore result
+                }
+
+                assertEquals(((ClickHouseResultSet) rs).getReadRows(), lines);
+            }
+        });
+    }
+}