Repository: milvus-io/milvus

Issue #31642: [Bug]: [restful v2]hybrid search failed with error `incomplete query result, missing id %!s(int64=3551), len(searchIDs) = 10, len(queryIDs) = 1, collection=448661016962138998: inconsistent requery result`
Created: 2024-03-27T07:25:46Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version:2.4-20240326-35664fa3-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior
[2024-03-27T02:43:44.707Z] {"code":2200,"message":"incomplete query result, missing id %!s(int64=515), len(searchIDs) = 10, len(queryIDs) = 1, collection=448661016962138998: inconsistent requery result"}

i am doing perf test for hybrid search restful API. 90% requests are failed with this error
[2024-03-27T02:43:53.440Z] Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
[2024-03-27T02:43:53.441Z] --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
[2024-03-27T02:43:53.441Z] POST     /v2/vectordb/entities/hybrid_search                                            14596 13169(90.22%) |    796     550    1115    800 |    9.30        8.60
[2024-03-27T02:43:53.441Z] --------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
[2024-03-27T02:43:53.441Z]          Aggregated                                                                     14596 13169(90.22%) |    796     550    1115    800 |    9.30        8.60

Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/restful_v2_perf_test/detail/restful_v2_perf_test/8/pipeline
pod info
[2024-03-27T02:48:22.999Z] restful-v2-perf-test-8-etcd-0                                     1/1     Running            0                  30m     10.104.15.160   4am-node20   <none>           <none>

[2024-03-27T02:48:22.999Z] restful-v2-perf-test-8-etcd-1                                     1/1     Running            0                  30m     10.104.31.103   4am-node34   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-etcd-2                                     1/1     Running            0                  30m     10.104.19.248   4am-node28   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-kafka-0                                    1/1     Running            2 (30m ago)        30m     10.104.15.158   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-kafka-1                                    1/1     Running            2 (30m ago)        30m     10.104.31.102   4am-node34   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-kafka-2                                    1/1     Running            2 (30m ago)        30m     10.104.27.42    4am-node31   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-milvus-standalone-b97575779-wqv5v          1/1     Running            1 (29m ago)        30m     10.104.15.157   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-minio-6b89664694-br2qf                     1/1     Running            0                  30m     10.104.15.159   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-t5w1t-6bgll-97lzm                          2/2     Running            0                  33m     10.104.13.31    4am-node16   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-zookeeper-0                                1/1     Running            0                  30m     10.104.15.156   4am-node20   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-zookeeper-1                                1/1     Running            0                  30m     10.104.27.37    4am-node31   <none>           <none>

[2024-03-27T02:48:23.000Z] restful-v2-perf-test-8-zookeeper-2                                1/1     Running            0                  30m     10.104.19.245   4am-node28   <none>           <none>

cluster: 4am
namespace: chaos-testing
Anything else?
related to #29656 (comment)

Duplicate issue #32225: [Bug]: [restful v2]hybrid search failed when nq is 2
Created: 2024-04-12T10:50:23Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version:2.4-20240412-d9892367-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior
[2024-04-12 18:45:00 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.32.80:19530/v2/vectordb/collections/create, 
cost time: 0.07076215744018555, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'b5e988b2-f8b9-11ee-973d-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_18_44_58_964494PEqCrMYL", "schema": {"autoId": true, "enableDynamicField": true, "fields": [{"fieldName": "book_id", "dataType": "Int64", "isPrimary": tr...at_vector_1", "indexName": "float_vector_1", "metricType": "COSINE"}, {"fieldName": "float_vector_2", "indexName": "float_vector_2", "metricType": "COSINE"}], "params": {"consistencyLevel": "Strong"}}, 
response: {"code":200,"data":{}} (milvus.py:17)
[2024-04-12 18:45:00 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.32.80:19530 (connectionpool.py:245)
[2024-04-12 18:45:00 - DEBUG - urllib3.connectionpool]: http://10.104.32.80:19530 "POST /v2/vectordb/collections/describe HTTP/1.1" 200 1333 (connectionpool.py:473)
[2024-04-12 18:45:00 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.32.80:19530/v2/vectordb/collections/describe, 
cost time: 0.03289198875427246, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'RequestId': 'b5f466b0-f8b9-11ee-973d-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_18_44_58_964494PEqCrMYL"}, 
response: {"code":200,"data":{"aliases":[],"autoId":true,"collectionID":449031326893805613,"collectionName":"test_collection_2024_04_12_18_44_58_964494PEqCrMYL","consistencyLevel":"Strong","description":"","ena...:"float_vector_1","metricType":"COSINE"},{"fieldName":"float_vector_2","indexName":"float_vector_2","metricType":"COSINE"}],"load":"LoadStateLoading","partitionsNum":64,"properties":[],"shardsNum":1}} (milvus.py:17)
[2024-04-12 18:45:00 - INFO - ci_test]: rsp: {'code': 200, 'data': {'aliases': [], 'autoId': True, 'collectionID': 449031326893805613, 'collectionName': 'test_collection_2024_04_12_18_44_58_964494PEqCrMYL', 'consistencyLevel': 'Strong', 'description': '', 'enableDynamicField': True, 'fields': [{'autoId': True, 'description': '', 'id': 100, 'name': 'book_id', 'partitionKey': False, 'primaryKey': True, 'type': 'Int64'}, {'autoId': False, 'description': '', 'id': 101, 'name': 'user_id', 'partitionKey': True, 'primaryKey': False, 'type': 'Int64'}, {'autoId': False, 'description': '', 'id': 102, 'name': 'word_count', 'partitionKey': False, 'primaryKey': False, 'type': 'Int64'}, {'autoId': False, 'description': '', 'id': 103, 'name': 'book_describe', 'params': [{'key': 'max_length', 'value': '256'}], 'partitionKey': False, 'primaryKey': False, 'type': 'VarChar'}, {'autoId': False, 'description': '', 'id': 104, 'name': 'float_vector_1', 'params': [{'key': 'dim', 'value': '128'}], 'partitionKey': False, 'primaryKey': False, 'type': 'FloatVector'}, {'autoId': False, 'description': '', 'id': 105, 'name': 'float_vector_2', 'params': [{'key': 'dim', 'value': '128'}], 'partitionKey': False, 'primaryKey': False, 'type': 'FloatVector'}], 'indexes': [{'fieldName': 'float_vector_1', 'indexName': 'float_vector_1', 'metricType': 'COSINE'}, {'fieldName': 'float_vector_2', 'indexName': 'float_vector_2', 'metricType': 'COSINE'}], 'load': 'LoadStateLoading', 'partitionsNum': 64, 'properties': [], 'shardsNum': 1}} (test_vector_operations.py:1164)
[2024-04-12 18:45:00 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.32.80:19530 (connectionpool.py:245)
[2024-04-12 18:45:04 - DEBUG - urllib3.connectionpool]: http://10.104.32.80:19530 "POST /v2/vectordb/entities/insert HTTP/1.1" 200 None (connectionpool.py:473)
[2024-04-12 18:45:04 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.32.80:19530/v2/vectordb/entities/insert, 
cost time: 3.274021863937378, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'Accept-Type-Allow-Int64': 'true', 'RequestId': 'b63a9f5e-f8b9-11ee-973d-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_18_44_58_964494PEqCrMYL", "data": [{"user_id": 0, "word_count": 0, "book_describe": "book_0", "float_vector_1": [0.12405901587826944, 0.09603262386960047... 0.09303034640802904, 0.10045834329727601, 0.03919269672992334, 0.04305713766553295, 0.02654005326722449, 0.07360782149643397, 0.13113123435853935, 0.026291956497121877], "dynamic_field_2999": 2999}]}, 
response: {"code":200,"data":{"insertCount":3000,"insertIds":[449031326893608595,449031326893608596,449031326893608597,449031326893608598,449031326893608599,449031326893608600,449031326893608601,449031326893608...3611584,449031326893611585,449031326893611586,449031326893611587,449031326893611588,449031326893611589,449031326893611590,449031326893611591,449031326893611592,449031326893611593,449031326893611594]}} (milvus.py:17)
[2024-04-12 18:45:05 - DEBUG - urllib3.connectionpool]: Starting new HTTP connection (1): 10.104.32.80:19530 (connectionpool.py:245)
[2024-04-12 18:45:06 - DEBUG - urllib3.connectionpool]: http://10.104.32.80:19530 "POST /v2/vectordb/entities/advanced_search HTTP/1.1" 200 294 (connectionpool.py:473)
[2024-04-12 18:45:06 - DEBUG - ci_test]: 
method: post, 
url: http://10.104.32.80:19530/v2/vectordb/entities/advanced_search, 
cost time: 0.8945760726928711, 
header: {'Content-Type': 'application/json', 'Authorization': 'Bearer root:Milvus', 'Accept-Type-Allow-Int64': 'true', 'RequestId': 'b90af9ea-f8b9-11ee-973d-acde48001122'}, 
payload: {"collectionName": "test_collection_2024_04_12_18_44_58_964494PEqCrMYL", "search": [{"data": [[0.06404256204912388, 0.03981077698932869, 0.008942506985435477, 0.005090076526972896, 0.015083862023994277, 0.11070310319768946, 0.04133312053735867, 0.06224479559979891, 0.03807799628647625, 0.1469795773257202, 0.1360158165107728, 0.12294938791950667, 0.062490716904511294, 0.0761823470813939, 0.04362815906019577, 0.09052342128579319, 0.12010991092635019, 0.015575898687186923, 0.023995703725737328, 0.02120012163805122, 0.1488273198319863, 0.05486243684461334, 0.04064333996507079, 0.14012940561729242, 0.05556161513838872, 0.023732890541570288, 0.13761311708539084, 0.03807202582628235, 0.061877096830785905, 0.0638772332384469, 0.03651400424800595, 0.15670142079153188, 0.08207341753142505, 0.1270902677124065, 0.010124877690596579, 0.08244132540648061, 0.10077136037052482, 0.1531155353062145, 0.12810225446697535, 0.07951598182929379, 0.016132589322135838, 0.13539912297853604, 0.020781963172937516...14, 0.10357021359795755, 0.09556662732640753, 0.05000998505857666, 0.0627565899660814, 0.021118427209994465, 0.13166422940627537, 0.033954066187043046, 0.12047377119163484, 0.08194310737701613, 0.07879951948717048, 0.06052198388148661, 0.12980152992877622, 0.02578534765487492, 0.135866647104243, 0.0528255656318728, 0.12682394037678807, 0.14412925015605063, 0.14105121024071582, 0.08388712182633686, 0.06412914527964793, 0.13623638277136954, 0.10669835913789934, 0.04183058820510128, 0.04791673182435017, 0.06202876306515234, 0.041158835544063334, 0.005933428543041745, 0.07931692946419754, 0.09923484200709566, 0.12117148533248119, 0.005319087086483033, 0.06324335273737865, 0.03431916736065866, 0.03480180921167877, 0.07590138903834519, 0.015257281107025785, 0.11117637628794531, 0.025666299873588812]], "annsField": "float_vector_2", "limit": 10, "outputFields": ["*"]}], "rerank": {"strategy": "rrf", "params": {"k": 10}}, "limit": 10, "outputFields": ["user_id", "word_count", "book_describe"]}, 
response: {"code":65535,"message":"fail to search on QueryNode 1: worker(1) query failed: Assert \"slice_nqs_prefix_sum_[num_slices_] == total_nq_\" at /go/src/github.com/milvus-io/milvus/internal/core/src/segcore/Reduce.cpp:42\n =\u003e illegal req sizes, slice_nqs_prefix_sum_[last] = 1, total_nq = 2"} (milvus.py:20)

it works well when nq is 1
    @pytest.mark.parametrize("insert_round", [1])
    @pytest.mark.parametrize("auto_id", [True])
    @pytest.mark.parametrize("is_partition_key", [True])
    @pytest.mark.parametrize("enable_dynamic_schema", [True])
    @pytest.mark.parametrize("nb", [3000])
    @pytest.mark.parametrize("dim", [128])
    @pytest.mark.parametrize("nq", [1, 2])
    def test_advanced_search_vector_with_multi_float32_vector_datatype(self, nb, dim, insert_round, auto_id,
                                                      is_partition_key, enable_dynamic_schema, nq):
        """
        Insert a vector with a simple payload
        """
        # create a collection
        name = gen_collection_name()
        payload = {
            "collectionName": name,
            "schema": {
                "autoId": auto_id,
                "enableDynamicField": enable_dynamic_schema,
                "fields": [
                    {"fieldName": "book_id", "dataType": "Int64", "isPrimary": True, "elementTypeParams": {}},
                    {"fieldName": "user_id", "dataType": "Int64", "isPartitionKey": is_partition_key,
                     "elementTypeParams": {}},
                    {"fieldName": "word_count", "dataType": "Int64", "elementTypeParams": {}},
                    {"fieldName": "book_describe", "dataType": "VarChar", "elementTypeParams": {"max_length": "256"}},
                    {"fieldName": "float_vector_1", "dataType": "FloatVector", "elementTypeParams": {"dim": f"{dim}"}},
                    {"fieldName": "float_vector_2", "dataType": "FloatVector", "elementTypeParams": {"dim": f"{dim}"}},
                ]
            },
            "indexParams": [
                {"fieldName": "float_vector_1", "indexName": "float_vector_1", "metricType": "COSINE"},
                {"fieldName": "float_vector_2", "indexName": "float_vector_2", "metricType": "COSINE"},

            ]
        }
        rsp = self.collection_client.collection_create(payload)
        assert rsp['code'] == 200
        rsp = self.collection_client.collection_describe(name)
        logger.info(f"rsp: {rsp}")
        assert rsp['code'] == 200
        # insert data
        for i in range(insert_round):
            data = []
            for i in range(nb):
                if auto_id:
                    tmp = {
                        "user_id": i%100,
                        "word_count": i,
                        "book_describe": f"book_{i}",
                        "float_vector_1": gen_vector(datatype="FloatVector", dim=dim),
                        "float_vector_2": gen_vector(datatype="FloatVector", dim=dim),
                    }
                else:
                    tmp = {
                        "book_id": i,
                        "user_id": i%100,
                        "word_count": i,
                        "book_describe": f"book_{i}",
                        "float_vector_1": gen_vector(datatype="FloatVector", dim=dim),
                        "float_vector_2": gen_vector(datatype="FloatVector", dim=dim),

                    }
                if enable_dynamic_schema:
                    tmp.update({f"dynamic_field_{i}": i})
                data.append(tmp)
            payload = {
                "collectionName": name,
                "data": data,
            }
            rsp = self.vector_client.vector_insert(payload)
            assert rsp['code'] == 200
            assert rsp['data']['insertCount'] == nb
        # advanced search data

        payload = {
            "collectionName": name,
            "search": [{
                "data": [gen_vector(datatype="FloatVector", dim=dim) for i in range(nq)],
                "annsField": "float_vector_1",
                "limit": 10,
                "outputFields": ["*"]
            },
                {
                "data": [gen_vector(datatype="FloatVector", dim=dim) for i in range(nq)],
                "annsField": "float_vector_2",
                "limit": 10,
                "outputFields": ["*"]
                }

            ],
            "rerank": {
                "strategy": "rrf",
                "params": {
                    "k": 10,
                }
            },
            "limit": 10,
            "outputFields": ["user_id", "word_count", "book_describe"]
        }

        rsp = self.vector_client.vector_advanced_search(payload)
        assert rsp['code'] == 200
        assert len(rsp['data']) == 10

Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
No response
Anything else?
No response

Closing comment (2024-04-17T02:13:34Z): related to #31642