Repository: ClickHouse/ClickHouse
Pull request #79350: feat: Add stringBytesUniq and stringBytesEntropy functions
Author: login: sachinkumarsingh092
Labels: name: pr-feature
description: Pull request with new product feature
name: manual approve
description: Manual approve required to run CI
name: can be tested
description: Allows running workflows for external contributors
name: pr-synced-to-cloud
description: The PR is synced to the cloud repo
Created: 2025-04-19T21:24:49Z | Merged: 2025-04-29T22:47:47Z

<!---
A technical comment, you are free to remove or leave it as it is when PR is created
The following categories are used in the next scripts, update them accordingly
utils/changelog/changelog.py
tests/ci/cancel_and_rerun_workflow_lambda/app.py
-->
### Changelog category (leave one):
- New Feature


### Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add `stringBytesUniq` and `stringBytesEntropy` functions to search for possibly random or encrypted data.

### Documentation entry for user-facing changes

- [x] Documentation is written (mandatory for new features)

<!---
Directly edit documentation source files in the "docs" folder with the same pull-request as code changes

or

Add a user-readable short description of the changes that should be added to docs.clickhouse.com below.

At a minimum, the following information should be added (but add more as needed).
- Motivation: Why is this function, table engine, etc. useful to ClickHouse users?

- Parameters: If the feature being added takes arguments, options or is influenced by settings, please list them below with a brief explanation.

- Example use: A query or command.
-->
Fixes https://github.com/ClickHouse/ClickHouse/issues/79305

**Changes:**
- Added the following two functions along with tests and docs for them:
  - `stringBytesUniq`: Counts the number of distinct bytes in a string.
  - `stringBytesEntropy` : Calculates Shannon's entropy of byte distribution in a string.



Diff:
diff --git a/ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt b/ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt
index 94e485cf38ec..7772e231977f 100644
--- a/ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt
+++ b/ci/jobs/scripts/check_style/aspell-ignore/en/aspell-dict.txt
@@ -2836,6 +2836,8 @@ stochasticlinearregression
 stochasticlogisticregression
 storages
 storig
+stringBytesEntropy
+stringBytesUniq
 stringCompare
 stringJaccardIndex
 stringJaccardIndexUTF
diff --git a/docs/en/sql-reference/functions/string-functions.md b/docs/en/sql-reference/functions/string-functions.md
index 207e551da385..9d3567111e6d 100644
--- a/docs/en/sql-reference/functions/string-functions.md
+++ b/docs/en/sql-reference/functions/string-functions.md
@@ -2637,3 +2637,67 @@ Result:
 1. │ [417784657,728683856,3071092609] │
    └──────────────────────────────────┘
 ```
+
+## stringBytesUniq {#stringbytesuniq}
+
+Counts the number of distinct bytes in a string.
+
+**Syntax**
+
+```sql
+stringBytesUniq(s)
+```
+
+**Arguments**
+
+- `s` — The string to analyze. [String](../data-types/string.md).
+
+**Returned value**
+
+- The number of distinct bytes in the string. [UInt16](../data-types/int-uint.md).
+
+**Example**
+
+```sql
+SELECT stringBytesUniq('Hello');
+```
+
+Result:
+
+```result
+┌─stringBytesUniq('Hello')─┐
+│                        4 │
+└──────────────────────────┘
+```
+
+## stringBytesEntropy {#stringbytesentropy}
+
+Calculates Shannon's entropy of byte distribution in a string.
+
+**Syntax**
+
+```sql
+stringBytesEntropy(s)
+```
+
+**Arguments**
+
+- `s` — The string to analyze. [String](../data-types/string.md).
+
+**Returned value**
+
+- Shannon's entropy of byte distribution in the string. [Float64](../data-types/float.md).
+
+**Example**
+
+```sql
+SELECT stringBytesEntropy('Hello, world!');
+```
+
+Result:
+
+```result
+┌─stringBytesEntropy('Hello, world!')─┐
+│                         3.07049960  │
+└─────────────────────────────────────┘
+```
diff --git a/src/Functions/stringBytes.h b/src/Functions/stringBytes.h
new file mode 100644
index 000000000000..ec949fbf1f2c
--- /dev/null
+++ b/src/Functions/stringBytes.h
@@ -0,0 +1,83 @@
+#pragma once
+
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+#include <cmath>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt16>)
+            return std::make_shared<DataTypeUInt16>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        size_t prev_offset = 0;
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const UInt8 * data_ptr = data.data() + prev_offset;
+            const size_t size = offsets[i] - prev_offset - 1;
+
+            vec_res[i] = Impl::process(data_ptr, size);
+            prev_offset = offsets[i];
+        }
+
+        return col_res;
+    }
+};
+
+}
diff --git a/src/Functions/stringBytesEntropy.cpp b/src/Functions/stringBytesEntropy.cpp
new file mode 100644
index 000000000000..cf8a407b2267
--- /dev/null
+++ b/src/Functions/stringBytesEntropy.cpp
@@ -0,0 +1,82 @@
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/IFunction.h>
+#include <Functions/stringBytes.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+#include <cmath>
+
+namespace DB
+{
+
+struct StringBytesEntropyImpl
+{
+    using ResultType = Float64;
+
+    static ResultType process(const UInt8 * data, size_t size)
+    {
+        if (size == 0)
+            return 0;
+
+        std::array<UInt32, 256> counters{};
+        const UInt8 * end = data + size;
+
+        for (; data < end; ++data)
+            counters[*data]++;
+
+        Float64 entropy = 0.0;
+
+        for (size_t byte = 0; byte < 256; ++byte)
+        {
+            UInt32 count = counters[byte];
+            if (count > 0)
+            {
+                Float64 p = static_cast<Float64>(count) / size;
+                entropy -= p * std::log2(p);
+            }
+        }
+
+        return entropy;
+    }
+};
+
+struct NameStringBytesEntropy
+{
+    static constexpr auto name = "stringBytesEntropy";
+};
+
+using FunctionStringBytesEntropy = FunctionStringBytes<StringBytesEntropyImpl, NameStringBytesEntropy>;
+
+REGISTER_FUNCTION(StringBytesEntropy)
+{
+    FunctionDocumentation::Description description = "Calculates Shannon's entropy of byte distribution in a string.";
+    FunctionDocumentation::Syntax syntax = "stringBytesEntropy(s);";
+    FunctionDocumentation::Arguments arguments = {
+        {"s", "The string to analyze. [String](../../sql-reference/data-types/string.md))"}
+    };
+    FunctionDocumentation::ReturnedValue returned_value = "The Shannon entropy of the byte distribution. [Float64](../../sql-reference/data-types/float.md).";
+    FunctionDocumentation::Examples examples = {
+        {"Example",
+         "SELECT stringBytesEntropy('Hello, world!');",
+         "3.180832987205441"}
+    };
+    FunctionDocumentation::Category category = FunctionDocumentation::Category::String;
+
+    FunctionDocumentation function_documentation = {
+        .description = description,
+        .syntax = syntax,
+        .arguments = arguments,
+        .returned_value = returned_value,
+        .examples = examples,
+        .category = category
+    };
+
+    factory.registerFunction<FunctionStringBytesEntropy>(function_documentation);
+}
+
+}
diff --git a/src/Functions/stringBytesUniq.cpp b/src/Functions/stringBytesUniq.cpp
new file mode 100644
index 000000000000..4f7a0cbe991e
--- /dev/null
+++ b/src/Functions/stringBytesUniq.cpp
@@ -0,0 +1,73 @@
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/stringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+#include <bit>
+#include <cmath>
+
+namespace DB
+{
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt16;
+
+    static ResultType process(const UInt8 * data, size_t size)
+    {
+        UInt64 mask[4] = {0};
+        const UInt8 * end = data + size;
+
+        for (; data < end; ++data)
+        {
+            UInt8 byte = *data;
+            mask[byte >> 6] |= (1ULL << (byte & 0x3F));
+        }
+
+        return std::popcount(mask[0]) + std::popcount(mask[1]) + std::popcount(mask[2]) + std::popcount(mask[3]);
+    }
+};
+
+
+struct NameStringBytesUniq
+{
+    static constexpr auto name = "stringBytesUniq";
+};
+
+
+using FunctionStringBytesUniq = FunctionStringBytes<StringBytesUniqImpl, NameStringBytesUniq>;
+
+REGISTER_FUNCTION(StringBytesUniq)
+{
+    FunctionDocumentation::Description description = "Counts the number of distinct bytes in a string.";
+    FunctionDocumentation::Syntax syntax = "stringBytesUniq(s);";
+    FunctionDocumentation::Arguments arguments = {
+        {"s", "The string to analyze. [String](../../sql-reference/data-types/string.md)"}
+    };
+    FunctionDocumentation::ReturnedValue returned_value = "The number of distinct bytes in the string. [UInt16](../../sql-reference/data-types/int-uint.md).";
+    FunctionDocumentation::Examples examples = {
+        {"Example",
+         "SELECT stringBytesUniq('Hello, world!');",
+         "10"}
+    };
+    FunctionDocumentation::Category category = FunctionDocumentation::Category::String;
+
+    FunctionDocumentation function_documentation = {
+        .description = description,
+        .syntax = syntax,
+        .arguments = arguments,
+        .returned_value = returned_value,
+        .examples = examples,
+        .category = category
+    };
+
+    factory.registerFunction<FunctionStringBytesUniq>(function_documentation);
+}
+
+}
diff --git a/tests/performance/string_bytes_functions.xml b/tests/performance/string_bytes_functions.xml
new file mode 100644
index 000000000000..00f73ba14137
--- /dev/null
+++ b/tests/performance/string_bytes_functions.xml
@@ -0,0 +1,11 @@
+<test>
+    <create_query>DROP TABLE IF EXISTS test_string_bytes</create_query>
+    <create_query>CREATE TABLE test_string_bytes(s String) ENGINE Memory</create_query>
+
+    <fill_query>INSERT INTO test_string_bytes SELECT reinterpretAsString(arrayJoin(range(0, 256))) FROM numbers(1000000)</fill_query>
+
+    <query>SELECT sum(stringBytesUniq(s)) FROM test_string_bytes FORMAT Null</query>
+    <query>SELECT sum(stringBytesEntropy(s)) FROM test_string_bytes FORMAT Null</query>
+
+    <drop_query>DROP TABLE IF EXISTS test_string_bytes</drop_query>
+</test>
diff --git a/tests/queries/0_stateless/03442_string_bytes_functions.reference b/tests/queries/0_stateless/03442_string_bytes_functions.reference
new file mode 100644
index 000000000000..c1b058237fd6
--- /dev/null
+++ b/tests/queries/0_stateless/03442_string_bytes_functions.reference
@@ -0,0 +1,86 @@
+-- { echo }
+
+SELECT stringBytesUniq('Hello');
+4
+SELECT stringBytesUniq('');
+0
+SELECT stringBytesUniq('aaaa');
+1
+SELECT stringBytesUniq('abcABC123');
+9
+SELECT stringBytesUniq(toNullable('Hello'));
+4
+SELECT stringBytesUniq(toLowCardinality('Hello'));
+4
+SELECT stringBytesEntropy('Hello');
+1.9219280948873623
+SELECT stringBytesEntropy('');
+0
+SELECT stringBytesEntropy('aaaa');
+0
+SELECT stringBytesEntropy('abcABC123');
+3.169925001442312
+SELECT stringBytesEntropy(toNullable('Hello'));
+1.9219280948873623
+SELECT stringBytesEntropy(toLowCardinality('Hello'));
+1.9219280948873623
+SELECT stringBytesUniq(NULL);
+\N
+SELECT stringBytesEntropy(NULL);
+\N
+SELECT stringBytesUniq(toNullable(NULL));
+\N
+SELECT stringBytesEntropy(toNullable(NULL));
+\N
+SELECT stringBytesUniq(unhex(concat(
+    '000102030405060708090A0B0C0D0E0F',
+    '101112131415161718191A1B1C1D1E1F',
+    '202122232425262728292A2B2C2D2E2F',
+    '303132333435363738393A3B3C3D3E3F',
+    '404142434445464748494A4B4C4D4E4F',
+    '505152535455565758595A5B5C5D5E5F',
+    '606162636465666768696A6B6C6D6E6F',
+    '707172737475767778797A7B7C7D7E7F',
+    '808182838485868788898A8B8C8D8E8F',
+    '909192939495969798999A9B9C9D9E9F',
+    'A0A1A2A3A4A5A6A7A8A9AAABACADAEAF',
+    'B0B1B2B3B4B5B6B7B8B9BABBBCBDBEBF',
+    'C0C1C2C3C4C5C6C7C8C9CACBCCCDCECF',
+    'D0D1D2D3D4D5D6D7D8D9DADBDCDDDEDF',
+    'E0E1E2E3E4E5E6E7E8E9EAEBECEDEEEF',
+    'F0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF'
+)));
+256
+SELECT stringBytesEntropy(unhex(concat(
+    '000102030405060708090A0B0C0D0E0F',
+    '101112131415161718191A1B1C1D1E1F',
+    '202122232425262728292A2B2C2D2E2F',
+    '303132333435363738393A3B3C3D3E3F',
+    '404142434445464748494A4B4C4D4E4F',
+    '505152535455565758595A5B5C5D5E5F',
+    '606162636465666768696A6B6C6D6E6F',
+    '707172737475767778797A7B7C7D7E7F',
+    '808182838485868788898A8B8C8D8E8F',
+    '909192939495969798999A9B9C9D9E9F',
+    'A0A1A2A3A4A5A6A7A8A9AAABACADAEAF',
+    'B0B1B2B3B4B5B6B7B8B9BABBBCBDBEBF',
+    'C0C1C2C3C4C5C6C7C8C9CACBCCCDCECF',
+    'D0D1D2D3D4D5D6D7D8D9DADBDCDDDEDF',
+    'E0E1E2E3E4E5E6E7E8E9EAEBECEDEEEF',
+    'F0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF'
+)));
+8
+SELECT stringBytesUniq(s) FROM (SELECT arrayJoin(['Hello', 'World', 'ClickHouse']) AS s);
+4
+5
+10
+SELECT stringBytesEntropy(s) FROM (SELECT arrayJoin(['Hello', 'World', 'ClickHouse']) AS s);
+1.9219280948873623
+2.321928094887362
+3.321928094887362
+SELECT arrayJoin(['Hello', 'World', 'Foo', 'Bar', 'ClickHouse']) AS str, stringBytesUniq(str), stringBytesEntropy(str);
+Hello	4	1.9219280948873623
+World	5	2.321928094887362
+Foo	2	0.9182958340544896
+Bar	3	1.584962500721156
+ClickHouse	10	3.321928094887362
diff --git a/tests/queries/0_stateless/03442_string_bytes_functions.sql b/tests/queries/0_stateless/03442_string_bytes_functions.sql
new file mode 100644
index 000000000000..a6ceafe16223
--- /dev/null
+++ b/tests/queries/0_stateless/03442_string_bytes_functions.sql
@@ -0,0 +1,61 @@
+-- { echo }
+
+SELECT stringBytesUniq('Hello');
+SELECT stringBytesUniq('');
+SELECT stringBytesUniq('aaaa');
+SELECT stringBytesUniq('abcABC123');
+SELECT stringBytesUniq(toNullable('Hello'));
+SELECT stringBytesUniq(toLowCardinality('Hello'));
+
+SELECT stringBytesEntropy('Hello');
+SELECT stringBytesEntropy('');
+SELECT stringBytesEntropy('aaaa');
+SELECT stringBytesEntropy('abcABC123');
+SELECT stringBytesEntropy(toNullable('Hello'));
+SELECT stringBytesEntropy(toLowCardinality('Hello'));
+
+SELECT stringBytesUniq(NULL);
+SELECT stringBytesEntropy(NULL);
+SELECT stringBytesUniq(toNullable(NULL));
+SELECT stringBytesEntropy(toNullable(NULL));
+
+SELECT stringBytesUniq(unhex(concat(
+    '000102030405060708090A0B0C0D0E0F',
+    '101112131415161718191A1B1C1D1E1F',
+    '202122232425262728292A2B2C2D2E2F',
+    '303132333435363738393A3B3C3D3E3F',
+    '404142434445464748494A4B4C4D4E4F',
+    '505152535455565758595A5B5C5D5E5F',
+    '606162636465666768696A6B6C6D6E6F',
+    '707172737475767778797A7B7C7D7E7F',
+    '808182838485868788898A8B8C8D8E8F',
+    '909192939495969798999A9B9C9D9E9F',
+    'A0A1A2A3A4A5A6A7A8A9AAABACADAEAF',
+    'B0B1B2B3B4B5B6B7B8B9BABBBCBDBEBF',
+    'C0C1C2C3C4C5C6C7C8C9CACBCCCDCECF',
+    'D0D1D2D3D4D5D6D7D8D9DADBDCDDDEDF',
+    'E0E1E2E3E4E5E6E7E8E9EAEBECEDEEEF',
+    'F0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF'
+)));
+SELECT stringBytesEntropy(unhex(concat(
+    '000102030405060708090A0B0C0D0E0F',
+    '101112131415161718191A1B1C1D1E1F',
+    '202122232425262728292A2B2C2D2E2F',
+    '303132333435363738393A3B3C3D3E3F',
+    '404142434445464748494A4B4C4D4E4F',
+    '505152535455565758595A5B5C5D5E5F',
+    '606162636465666768696A6B6C6D6E6F',
+    '707172737475767778797A7B7C7D7E7F',
+    '808182838485868788898A8B8C8D8E8F',
+    '909192939495969798999A9B9C9D9E9F',
+    'A0A1A2A3A4A5A6A7A8A9AAABACADAEAF',
+    'B0B1B2B3B4B5B6B7B8B9BABBBCBDBEBF',
+    'C0C1C2C3C4C5C6C7C8C9CACBCCCDCECF',
+    'D0D1D2D3D4D5D6D7D8D9DADBDCDDDEDF',
+    'E0E1E2E3E4E5E6E7E8E9EAEBECEDEEEF',
+    'F0F1F2F3F4F5F6F7F8F9FAFBFCFDFEFF'
+)));
+
+SELECT stringBytesUniq(s) FROM (SELECT arrayJoin(['Hello', 'World', 'ClickHouse']) AS s);
+SELECT stringBytesEntropy(s) FROM (SELECT arrayJoin(['Hello', 'World', 'ClickHouse']) AS s);
+SELECT arrayJoin(['Hello', 'World', 'Foo', 'Bar', 'ClickHouse']) AS str, stringBytesUniq(str), stringBytesEntropy(str);
\ No newline at end of file


Reviews:
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:33:20Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:34:05Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:34:33Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:35:05Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:35:42Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:36:39Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:37:33Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:38:17Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:39:04Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:39:22Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:39:57Z
user: login: alexey-milovidov
body: Add a performance test as well (tests/performance).
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:40:14Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T03:42:44Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:15:54Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:16:14Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:16:27Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:18:42Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:19:40Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:51:32Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:51:56Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T18:52:01Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T19:08:43Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T19:08:47Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T19:34:55Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-20T20:35:16Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T21:43:15Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T05:45:04Z
user: login: alexey-milovidov
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T05:45:53Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T06:30:19Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-21T06:33:43Z
user: login: alexey-milovidov
body: Looks good!

One ask - could you please compare the performance (on your machine - ok) with and without the optimization with current_generation?

You can do it on the https://github.com/ClickHouse/ClickBench dataset, on the URL, SearchPhrase, Title, MobilePhoneModel columns.
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-21T20:31:46Z
user: login: bharatnc
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-28T20:35:41Z
user: login: sachinkumarsingh092
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T08:01:55Z
user: login: alexey-milovidov
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-29T22:47:39Z

Review comments:
diff_hunk: @@ -2637,3 +2637,67 @@ Result:
 1. │ [417784657,728683856,3071092609] │
    └──────────────────────────────────┘
 ```
+
+## stringBytesUniq {#stringbytesuniq}
+
+Counts the number of distinct bytes in a string.
+
+**Syntax**
+
+```sql
+stringBytesUniq(s)
+```
+
+**Arguments**
+
+- `s` — The string to analyze. [String](../data-types/string.md).
+
+**Returned value**
+
+- The number of distinct bytes in the string. [UInt8](../data-types/int-uint.md).
path: docs/en/sql-reference/functions/string-functions.md
user: login: alexey-milovidov
body: The number of distinct bytes can range from 0 to 256, while the range of the UInt8 data type is just 0 to 255. Let's change to UInt16.
created_at: 2025-04-20T03:33:20Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: We only use clang, so there is no problem if we keep only this branch.
created_at: 2025-04-20T03:34:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: There is `std::popcount`
created_at: 2025-04-20T03:34:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
+    return __builtin_popcountll(x);
+#else
+    UInt8 count = 0;
+    while (x)
+    {
+        count += x & 1;
+        x >>= 1;
+    }
+    return count;
+#endif
+}
+
+class ByteCounters
+{
+private:
+    static constexpr size_t COUNTERS_SIZE = 256;
+    UInt32 counters[COUNTERS_SIZE] = {0};
+    UInt32 current_generation = 0;
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: Btw, was this optimization worth it?
created_at: 2025-04-20T03:35:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
+    return __builtin_popcountll(x);
+#else
+    UInt8 count = 0;
+    while (x)
+    {
+        count += x & 1;
+        x >>= 1;
+    }
+    return count;
+#endif
+}
+
+class ByteCounters
+{
+private:
+    static constexpr size_t COUNTERS_SIZE = 256;
+    UInt32 counters[COUNTERS_SIZE] = {0};
+    UInt32 current_generation = 0;
+    UInt32 generation_mask = 0x80000000;
+    size_t total_count = 0;
+
+public:
+    void add(UInt8 byte)
+    {
+        UInt32 & counter = counters[byte];
+        if ((counter & generation_mask) != current_generation)
+            counter = current_generation;
+        counter++;
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: A minor thing: we write prefix increment by default.
created_at: 2025-04-20T03:35:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
+    return __builtin_popcountll(x);
+#else
+    UInt8 count = 0;
+    while (x)
+    {
+        count += x & 1;
+        x >>= 1;
+    }
+    return count;
+#endif
+}
+
+class ByteCounters
+{
+private:
+    static constexpr size_t COUNTERS_SIZE = 256;
+    UInt32 counters[COUNTERS_SIZE] = {0};
+    UInt32 current_generation = 0;
+    UInt32 generation_mask = 0x80000000;
+    size_t total_count = 0;
+
+public:
+    void add(UInt8 byte)
+    {
+        UInt32 & counter = counters[byte];
+        if ((counter & generation_mask) != current_generation)
+            counter = current_generation;
+        counter++;
+        total_count++;
+    }
+
+    void nextString()
+    {
+        current_generation = current_generation ? 0 : generation_mask;
+        total_count = 0;
+    }
+
+    UInt32 get(UInt8 byte) const
+    {
+        UInt32 counter = counters[byte];
+        if ((counter & generation_mask) != current_generation)
+            return 0;
+        return counter & ~generation_mask;
+    }
+
+    size_t getTotalCount() const { return total_count; }
+};
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt8;
+
+    static ResultType process(const char * str, size_t size)
+    {
+        uint64_t mask[4] = {0};
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: It's unusual, that UInt64 is used elsewhere, but uint64_t only here.
created_at: 2025-04-20T03:36:38Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,80 @@
+#pragma once
+
+#include <cmath>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt8>)
+            return std::make_shared<DataTypeUInt8>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const char * str = reinterpret_cast<const char *>(data.data() + (i == 0 ? 0 : offsets[i - 1]));
path: src/Functions/FunctionsStringBytes.h
user: login: alexey-milovidov
body: I see we do `reinterpret_cast` twice: here to char *  and there back to UInt8 *. We can remove both.
created_at: 2025-04-20T03:37:33Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,80 @@
+#pragma once
+
+#include <cmath>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt8>)
+            return std::make_shared<DataTypeUInt8>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const char * str = reinterpret_cast<const char *>(data.data() + (i == 0 ? 0 : offsets[i - 1]));
path: src/Functions/FunctionsStringBytes.h
user: login: alexey-milovidov
body: There is one unusual thing about offsets (PaddedPODArray) - they allow referencing -1th element, so it is safe to always write `offsets[i - 1]`
created_at: 2025-04-20T03:38:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,80 @@
+#pragma once
+
+#include <cmath>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt8>)
+            return std::make_shared<DataTypeUInt8>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const char * str = reinterpret_cast<const char *>(data.data() + (i == 0 ? 0 : offsets[i - 1]));
+            const size_t size = offsets[i] - (i == 0 ? 0 : offsets[i - 1]) - 1;
path: src/Functions/FunctionsStringBytes.h
user: login: alexey-milovidov
body: However, idiomatically is to keep the `prev_offset` variable and change it in the loop.
created_at: 2025-04-20T03:39:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,18 @@
+SELECT stringBytesUniq('Hello');
path: tests/queries/0_stateless/03442_string_bytes_functions.sql
user: login: alexey-milovidov
body: At least one test with all 256 bytes.
created_at: 2025-04-20T03:39:22Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,18 @@
+SELECT stringBytesUniq('Hello');
+SELECT stringBytesUniq('');
+SELECT stringBytesUniq('aaaa');
+SELECT stringBytesUniq('abcABC123');
+SELECT stringBytesUniq(toNullable('Hello'));
+SELECT stringBytesUniq(toLowCardinality('Hello'));
+
+SELECT stringBytesEntropy('Hello');
+SELECT stringBytesEntropy('');
+SELECT stringBytesEntropy('aaaa');
+SELECT stringBytesEntropy('abcABC123');
+SELECT stringBytesEntropy(toNullable('Hello'));
+SELECT stringBytesEntropy(toLowCardinality('Hello'));
+
path: tests/queries/0_stateless/03442_string_bytes_functions.sql
user: login: alexey-milovidov
body: At least one test when the string is non constant (there is more than one string in a block).
created_at: 2025-04-20T03:39:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: We prefer every function in a separate file, as it opens possibilities to, e.g., link to a file from the docs, provide more precise authorship info, etc. So let's split into two files.

And then the .h file will be unneeded, so we can keep everything inside two .cpp files.

These .cpp files should be named exactly as the function, e.g., `stringBytesUniq.cpp`
created_at: 2025-04-20T03:42:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
+    return __builtin_popcountll(x);
+#else
+    UInt8 count = 0;
+    while (x)
+    {
+        count += x & 1;
+        x >>= 1;
+    }
+    return count;
+#endif
+}
+
+class ByteCounters
+{
+private:
+    static constexpr size_t COUNTERS_SIZE = 256;
+    UInt32 counters[COUNTERS_SIZE] = {0};
+    UInt32 current_generation = 0;
path: src/Functions/FunctionsStringBytes.cpp
user: login: sachinkumarsingh092
body: I didn't benchmark it specifically, but it definitely seems to be worth it because instead of resetting all 256 counters (O(n)) between strings, it uses the bespoke generation counter, which is an O(1) operation. For columns with many sequential strings, this approach will be significantly faster due to lower cost of initialisation for the individual strings.
created_at: 2025-04-20T18:15:54Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
path: src/Functions/FunctionsStringBytes.cpp
user: login: sachinkumarsingh092
body: TIL about this in C++20. Using `std::popcount` instead.
created_at: 2025-04-20T18:16:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -2637,3 +2637,67 @@ Result:
 1. │ [417784657,728683856,3071092609] │
    └──────────────────────────────────┘
 ```
+
+## stringBytesUniq {#stringbytesuniq}
+
+Counts the number of distinct bytes in a string.
+
+**Syntax**
+
+```sql
+stringBytesUniq(s)
+```
+
+**Arguments**
+
+- `s` — The string to analyze. [String](../data-types/string.md).
+
+**Returned value**
+
+- The number of distinct bytes in the string. [UInt8](../data-types/int-uint.md).
path: docs/en/sql-reference/functions/string-functions.md
user: login: sachinkumarsingh092
body: Done.
created_at: 2025-04-20T18:16:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
+    return __builtin_popcountll(x);
+#else
+    UInt8 count = 0;
+    while (x)
+    {
+        count += x & 1;
+        x >>= 1;
+    }
+    return count;
+#endif
+}
+
+class ByteCounters
+{
+private:
+    static constexpr size_t COUNTERS_SIZE = 256;
+    UInt32 counters[COUNTERS_SIZE] = {0};
+    UInt32 current_generation = 0;
+    UInt32 generation_mask = 0x80000000;
+    size_t total_count = 0;
+
+public:
+    void add(UInt8 byte)
+    {
+        UInt32 & counter = counters[byte];
+        if ((counter & generation_mask) != current_generation)
+            counter = current_generation;
+        counter++;
path: src/Functions/FunctionsStringBytes.cpp
user: login: sachinkumarsingh092
body: Fixed.
created_at: 2025-04-20T18:18:42Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/FunctionsStringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+namespace DB
+{
+
+inline UInt8 countBits(UInt64 x)
+{
+#if defined(__clang__)
+    return __builtin_popcountll(x);
+#else
+    UInt8 count = 0;
+    while (x)
+    {
+        count += x & 1;
+        x >>= 1;
+    }
+    return count;
+#endif
+}
+
+class ByteCounters
+{
+private:
+    static constexpr size_t COUNTERS_SIZE = 256;
+    UInt32 counters[COUNTERS_SIZE] = {0};
+    UInt32 current_generation = 0;
+    UInt32 generation_mask = 0x80000000;
+    size_t total_count = 0;
+
+public:
+    void add(UInt8 byte)
+    {
+        UInt32 & counter = counters[byte];
+        if ((counter & generation_mask) != current_generation)
+            counter = current_generation;
+        counter++;
+        total_count++;
+    }
+
+    void nextString()
+    {
+        current_generation = current_generation ? 0 : generation_mask;
+        total_count = 0;
+    }
+
+    UInt32 get(UInt8 byte) const
+    {
+        UInt32 counter = counters[byte];
+        if ((counter & generation_mask) != current_generation)
+            return 0;
+        return counter & ~generation_mask;
+    }
+
+    size_t getTotalCount() const { return total_count; }
+};
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt8;
+
+    static ResultType process(const char * str, size_t size)
+    {
+        uint64_t mask[4] = {0};
path: src/Functions/FunctionsStringBytes.cpp
user: login: sachinkumarsingh092
body: Copy-pasta issue. Fixed.
created_at: 2025-04-20T18:19:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,80 @@
+#pragma once
+
+#include <cmath>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt8>)
+            return std::make_shared<DataTypeUInt8>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const char * str = reinterpret_cast<const char *>(data.data() + (i == 0 ? 0 : offsets[i - 1]));
path: src/Functions/FunctionsStringBytes.h
user: login: sachinkumarsingh092
body: Fair. I changed the processing function to take `const UInt8 *` to simply things.

created_at: 2025-04-20T18:51:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,80 @@
+#pragma once
+
+#include <cmath>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt8>)
+            return std::make_shared<DataTypeUInt8>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const char * str = reinterpret_cast<const char *>(data.data() + (i == 0 ? 0 : offsets[i - 1]));
path: src/Functions/FunctionsStringBytes.h
user: login: sachinkumarsingh092
body: I'd love to read how that works. For now I fixed it.
created_at: 2025-04-20T18:51:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,80 @@
+#pragma once
+
+#include <cmath>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+
+namespace DB
+{
+
+namespace ErrorCodes
+{
+extern const int ILLEGAL_TYPE_OF_ARGUMENT;
+extern const int ILLEGAL_COLUMN;
+}
+
+template <typename Impl, typename Name>
+class FunctionStringBytes : public IFunction
+{
+public:
+    static constexpr auto name = Name::name;
+    using ResultType = typename Impl::ResultType;
+
+    static FunctionPtr create(ContextPtr) { return std::make_shared<FunctionStringBytes>(); }
+
+    String getName() const override { return name; }
+
+    size_t getNumberOfArguments() const override { return 1; }
+
+    bool useDefaultImplementationForConstants() const override { return true; }
+
+    bool isSuitableForShortCircuitArgumentsExecution(const DataTypesWithConstInfo &) const override { return true; }
+
+    DataTypePtr getReturnTypeImpl(const ColumnsWithTypeAndName & arguments) const override
+    {
+        if (!isString(arguments[0].type))
+            throw Exception(
+                ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT,
+                "Illegal type {} of argument of function {}",
+                arguments[0].type->getName(),
+                getName());
+
+        if constexpr (std::is_same_v<ResultType, UInt8>)
+            return std::make_shared<DataTypeUInt8>();
+        else if constexpr (std::is_same_v<ResultType, Float64>)
+            return std::make_shared<DataTypeFloat64>();
+    }
+
+    ColumnPtr executeImpl(const ColumnsWithTypeAndName & arguments, const DataTypePtr &, size_t input_rows_count) const override
+    {
+        const ColumnPtr column = arguments[0].column;
+        const ColumnString * col_str = checkAndGetColumn<ColumnString>(column.get());
+
+        if (!col_str)
+            throw Exception(
+                ErrorCodes::ILLEGAL_COLUMN, "Illegal column {} of argument of function {}", arguments[0].column->getName(), getName());
+
+        auto col_res = ColumnVector<ResultType>::create();
+        auto & vec_res = col_res->getData();
+        vec_res.resize(input_rows_count);
+
+        const ColumnString::Chars & data = col_str->getChars();
+        const ColumnString::Offsets & offsets = col_str->getOffsets();
+
+        for (size_t i = 0; i < input_rows_count; ++i)
+        {
+            const char * str = reinterpret_cast<const char *>(data.data() + (i == 0 ? 0 : offsets[i - 1]));
+            const size_t size = offsets[i] - (i == 0 ? 0 : offsets[i - 1]) - 1;
path: src/Functions/FunctionsStringBytes.h
user: login: sachinkumarsingh092
body: Done.
created_at: 2025-04-20T18:52:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,18 @@
+SELECT stringBytesUniq('Hello');
path: tests/queries/0_stateless/03442_string_bytes_functions.sql
user: login: sachinkumarsingh092
body: Added.
created_at: 2025-04-20T19:08:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,18 @@
+SELECT stringBytesUniq('Hello');
+SELECT stringBytesUniq('');
+SELECT stringBytesUniq('aaaa');
+SELECT stringBytesUniq('abcABC123');
+SELECT stringBytesUniq(toNullable('Hello'));
+SELECT stringBytesUniq(toLowCardinality('Hello'));
+
+SELECT stringBytesEntropy('Hello');
+SELECT stringBytesEntropy('');
+SELECT stringBytesEntropy('aaaa');
+SELECT stringBytesEntropy('abcABC123');
+SELECT stringBytesEntropy(toNullable('Hello'));
+SELECT stringBytesEntropy(toLowCardinality('Hello'));
+
path: tests/queries/0_stateless/03442_string_bytes_functions.sql
user: login: sachinkumarsingh092
body: Added.
created_at: 2025-04-20T19:08:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
path: src/Functions/FunctionsStringBytes.cpp
user: login: sachinkumarsingh092
body: I understand it will be good to separate the core logic of the function in individual .cpp files, but wouldn't we still need a common .h file to re-use the `class FunctionStringBytes` for processing? Wouldn't re-implementing them for individual program in .cpp files lead to code repetition? 
Sorry if I misunderstood something.
created_at: 2025-04-20T19:34:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
path: src/Functions/FunctionsStringBytes.cpp
user: login: alexey-milovidov
body: Yes, you are right. Let's keep FunctionStringBytes in the header file. The .cpp files will contain the corresponding Impl, instantiate the template, and register the function.
created_at: 2025-04-20T20:35:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,142 @@
+#include <cmath>
path: src/Functions/FunctionsStringBytes.cpp
user: login: sachinkumarsingh092
body: Made this change as well.
created_at: 2025-04-20T21:43:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,29 @@
+4
+0
+1
+9
+4
+4
+1.9219280948873623
+0
+0
+3.169925001442312
+1.9219280948873623
+1.9219280948873623
+\N
+\N
+\N
+\N
+0
path: tests/queries/0_stateless/03442_string_bytes_functions.reference
user: login: alexey-milovidov
body: This result (0) looks incorrect.
created_at: 2025-04-21T05:45:04Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,52 @@
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/stringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+#include <bit>
+#include <cmath>
+
+namespace DB
+{
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt8;
path: src/Functions/stringBytesUniq.cpp
user: login: alexey-milovidov
body: Because the type here wasn't changed.
created_at: 2025-04-21T05:45:53Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,29 @@
+4
+0
+1
+9
+4
+4
+1.9219280948873623
+0
+0
+3.169925001442312
+1.9219280948873623
+1.9219280948873623
+\N
+\N
+\N
+\N
+0
path: tests/queries/0_stateless/03442_string_bytes_functions.reference
user: login: sachinkumarsingh092
body: Ah yes, corner case. I now use UInt16.
created_at: 2025-04-21T06:30:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/stringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+#include <bit>
+#include <cmath>
+
+namespace DB
+{
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt8;
path: src/Functions/stringBytesUniq.cpp
user: login: sachinkumarsingh092
body: I now use UInt16.
created_at: 2025-04-21T06:33:43Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/stringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+#include <bit>
+#include <cmath>
+
+namespace DB
+{
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt16;
+
+    static ResultType process(const UInt8 * data, size_t size)
+    {
+        UInt64 mask[4] = {0};
+        const UInt8 * end = data + size;
+
+        for (; data < end; ++data)
+        {
+            UInt8 byte = *data;
+            mask[byte >> 6] |= (1ULL << (byte & 0x3F));
+        }
+
+        return std::popcount(mask[0]) + std::popcount(mask[1]) + std::popcount(mask[2]) + std::popcount(mask[3]);
+    }
+};
+
+
+struct NameStringBytesUniq
+{
+    static constexpr auto name = "stringBytesUniq";
+};
+
+
+using FunctionStringBytesUniq = FunctionStringBytes<StringBytesUniqImpl, NameStringBytesUniq>;
+
+REGISTER_FUNCTION(StringBytesUniq)
+{
+    factory.registerFunction<FunctionStringBytesUniq>(
+        FunctionDocumentation{.description = R"(Counts the number of distinct bytes in a string.)"});
path: src/Functions/stringBytesUniq.cpp
user: login: bharatnc
body: Actually, the docs build failure does look related. Please provide the complete in code documentation as described the README for both the functions: https://github.com/ClickHouse/ClickHouse/blob/e916901443b1b1eeddbdcf9de48adc9cd62cf6a5/src/Functions/README.md. Currently the docs check are failing because of that. You can look at other similar functions for example.
```
/home/ubuntu/actions-runner/_work/ClickHouse/ClickHouse/src/Functions/stringBytesUniq.cpp:49:99: error: missing field 'category' initializer [-Werror,-Wmissing-designated-field-initializers]
           49 |         FunctionDocumentation{.description = R"(Counts the number of distinct bytes in a string.)"});
              |                                                                                                   ^
        1 error generated.
```
created_at: 2025-04-28T20:35:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,52 @@
+#include <Columns/ColumnConst.h>
+#include <Columns/ColumnString.h>
+#include <Columns/ColumnVector.h>
+#include <DataTypes/DataTypeString.h>
+#include <DataTypes/DataTypesNumber.h>
+#include <Functions/FunctionFactory.h>
+#include <Functions/stringBytes.h>
+#include <Functions/IFunction.h>
+#include <Common/BitHelpers.h>
+#include <Common/PODArray.h>
+
+#include <bit>
+#include <cmath>
+
+namespace DB
+{
+
+struct StringBytesUniqImpl
+{
+    using ResultType = UInt16;
+
+    static ResultType process(const UInt8 * data, size_t size)
+    {
+        UInt64 mask[4] = {0};
+        const UInt8 * end = data + size;
+
+        for (; data < end; ++data)
+        {
+            UInt8 byte = *data;
+            mask[byte >> 6] |= (1ULL << (byte & 0x3F));
+        }
+
+        return std::popcount(mask[0]) + std::popcount(mask[1]) + std::popcount(mask[2]) + std::popcount(mask[3]);
+    }
+};
+
+
+struct NameStringBytesUniq
+{
+    static constexpr auto name = "stringBytesUniq";
+};
+
+
+using FunctionStringBytesUniq = FunctionStringBytes<StringBytesUniqImpl, NameStringBytesUniq>;
+
+REGISTER_FUNCTION(StringBytesUniq)
+{
+    factory.registerFunction<FunctionStringBytesUniq>(
+        FunctionDocumentation{.description = R"(Counts the number of distinct bytes in a string.)"});
path: src/Functions/stringBytesUniq.cpp
user: login: sachinkumarsingh092
body: Fixed in https://github.com/ClickHouse/ClickHouse/pull/79350/commits/7e26ad41c29cc02d6185cd7d523b4f111ab862ed
created_at: 2025-04-29T08:01:55Z
author_association: CONTRIBUTOR

PR comments:
user: login: CLAassistant
created_at: 2025-04-19T21:24:55Z
body: [![CLA assistant check](https://cla-assistant.io/pull/badge/signed)](https://cla-assistant.io/ClickHouse/ClickHouse?pullRequest=79350) <br/>All committers have signed the CLA.
author_association: NONE
user: login: clickhouse-gh[bot]
created_at: 2025-04-20T00:28:21Z
body: Workflow [[PR](https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=79350&sha=latest&name_0=PR)], commit [7e26ad41]
author_association: CONTRIBUTOR
user: login: alexey-milovidov
created_at: 2025-04-20T00:35:23Z
body: At first glance, the code looks good :muscle:
Let's see what CI will tell...
author_association: MEMBER
user: login: alexey-milovidov
created_at: 2025-04-20T03:31:25Z
body: > Logical error: 'Can't set alias of * of Asterisk'.

This is unrelated.
author_association: MEMBER
user: login: sachinkumarsingh092
created_at: 2025-04-21T04:39:12Z
body: The integration tests failure seems to be unrelated to the changes; related to `test_make_clone_covered_by_broken_detached_dir_exists`, `test_adding_replica`, and `test_limited_fetches_for_server` in different integration tests.
The stateless tests also seemed to be unrelate. 
I could be wrong but these all seems to be an intermittent issues. Can we rerun these failing tests?
author_association: CONTRIBUTOR
user: login: sachinkumarsingh092
created_at: 2025-04-22T21:56:28Z
body: Hi @alexey-milovidov, I did some benchmarking.
So I hacked-patched the `benchmark.sh` script from `ClickBench` but substituted `run.sh` inside it to form a consistent script. This is the script I finally used:
```bash
#!/bin/bash -e

./programs/clickhouse-client < create.sql

./programs/clickhouse-client --query "INSERT INTO hits SELECT * FROM url('https://clickhouse-public-datasets.s3.amazonaws.com/hits_compatible/hits.tsv.gz')" --time

TRIES=3
QUERY_NUM=1
cat queries.sql | while read -r query; do
    echo -n "["
    for i in $(seq 1 $TRIES); do
        (./programs/clickhouse-client --time --format=Null --query="$query" --progress 0 ${EXTRA_SETTINGS} 2>&1 |
            grep -Eo '^[0-9]+\.[0-9]+$' || echo -n "null") | tr -d '\n'

        [[ "$i" != $TRIES ]] && echo -n ", "
    done
    echo "],"

    QUERY_NUM=$((QUERY_NUM + 1))
done

./programs/clickhouse-client --query "SELECT total_bytes FROM system.tables WHERE name = 'hits' AND database = 'default'"
```
Here `./programs/clickhouse-client ` is the locally built client, `create.sql` creates the hits table and queries.sql looks like this:
```
SELECT stringBytesUniq(URL), stringBytesEntropy(URL) FROM hits;
SELECT stringBytesUniq(SearchPhrase), stringBytesEntropy(SearchPhrase) FROM hits;
SELECT stringBytesUniq(Title), stringBytesEntropy(Title) FROM hits;
SELECT stringBytesUniq(MobilePhoneModel), stringBytesEntropy(MobilePhoneModel) FROM hits;
```

I then switch b/w optimised and un-optimised version by re-building the server. Here's the implementation I used for the un-optimised version:
```cpp
   static ResultType process(const UInt8 * data, size_t size)
    {
        if (size == 0)
            return 0;

        std::array<UInt32, 256> counters{};
        const UInt8 * end = data + size;

        // Count frequency of each byte
        for (; data < end; ++data)
            counters[*data]++;

        Float64 entropy = 0.0;
        Float64 total = size;

        for (size_t byte = 0; byte < 256; ++byte)
        {
            UInt32 count = counters[byte];
            if (count > 0)
            {
                double p = static_cast<double>(count) / total;
                entropy -= p * std::log2(p);
            }
        }

        return entropy;
    }
```

And also, instead of inserting the entire table, I just used 40million rows:
```
SELECT count(*)
FROM hits

Query id: cdf12fe1-046d-4936-b6b5-b93760c0e2b8

   ┌──count()─┐
1. │ 40213741 │ -- 40.21 million
   └──────────┘
 ```

Having done all that, here are the results of optimised and unoptimised version for 3 runs:
```bash
# unoptimised version
~/.../build/benchmark >>> ./benchmark.sh                                                                        
[2.136, 2.115, 2.284],
[0.240, 0.240, 0.225],
[2.246, 2.026, 2.088],
[0.076, 0.074, 0.078],
5018654101
~/.../build/benchmark >>> ./benchmark.sh                                                                      
[2.276, 1.969, 2.866],
[0.308, 0.428, 0.275],
[3.662, 3.054, 2.660],
[0.103, 0.122, 0.082],
5018654101
~/.../build/benchmark >>> ./benchmark.sh                                                                        
[2.228, 2.071, 2.104],
[0.256, 0.236, 0.234],
[2.329, 2.192, 2.645],
[0.088, 0.082, 0.082],
5018654101
                                                       
# optimised version
~/.../build/benchmark >>> ./benchmark.sh                                                                       
[2.592, 2.198, 2.237],
[0.287, 0.237, 0.256],
[2.443, 2.304, 2.327],
[0.089, 0.080, 0.085],
5018654101
~/.../build/benchmark >>> ./benchmark.sh                                                                        
[2.333, 2.151, 2.251],
[0.277, 0.298, 0.274],
[2.410, 2.303, 2.243],
[0.084, 0.081, 0.082],
5018654101
~/.../build/benchmark >>> ./benchmark.sh                                                                       
[2.270, 2.391, 2.267],
[0.284, 0.243, 0.246],
[2.422, 2.298, 2.300],
[0.082, 0.084, 0.083],
5018654101
```

Seems like the unoptimised version performed better!!?? Not sure if I did something wrong in the entire setup.
author_association: CONTRIBUTOR
user: login: alexey-milovidov
created_at: 2025-04-23T04:53:35Z
body: Yes, it was suspected because the amount of memory to clear is quite small, but the optimization introduces branches in the code, and then it is more difficult for the compiler to vectorize.

Let's switch to the unoptimized version, and we can merge!
author_association: MEMBER
user: login: sachinkumarsingh092
created_at: 2025-04-24T05:48:31Z
body: @alexey-milovidov can we retest?
author_association: CONTRIBUTOR
user: login: sachinkumarsingh092
created_at: 2025-04-28T07:58:40Z
body: Not sure if I'm looking into it right, but the failures seems unrelated?
author_association: CONTRIBUTOR
user: login: bharatnc
created_at: 2025-04-28T20:07:30Z
body: > Not sure if I'm looking into it right, but the failures seems unrelated?

Failures are unrelated. CI for the PR anyways seems broken so I tried merging master into this branch to try fix some of these unrelated failures.
author_association: CONTRIBUTOR