Repository: milvus-io/milvus

Issue #27012: [Bug]: Many operations failed with error `syncTimestamp Failed` after reinstallation and standalone pod failure chaos
Created: 2023-09-12T03:12:11Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version:2.3.0-20230911-57bed4a0
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):rocksmq    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior

[2023-09-11T12:54:54.495Z] <name>: deploy_test_index_type_HNSW_is_compacted_not_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_1_is_deleted_is_deleted_data_size_3000

[2023-09-11T12:54:54.495Z] <description>: 

[2023-09-11T12:54:54.496Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'descri......  (api_request.py:37)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:07 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 120} (api_request.py:62)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:10 - DEBUG - ci_test]: (api_response) : None  (api_request.py:37)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:10 - INFO - ci_test]: inserted 3000 data into collection deploy_test_index_type_HNSW_is_compacted_not_compacted_segment_status_all_is_string_indexed_is_string_indexed_replica_number_1_is_deleted_is_deleted_data_size_3000 (common_func.py:1224)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:10 - DEBUG - ci_test]: Dynamic field is enabled: False (common_func.py:1226)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:10 - DEBUG - ci_test]: (api_request)  : [Collection.insert] args: [      int64   float varchar                                       float_vector

[2023-09-11T12:54:54.496Z] 0         0     0.0       0  [0.008101348437231256, 0.061506697567280735, 0...

[2023-09-11T12:54:54.496Z] 1         1     1.0       1  [0.011208385715317033, 0.047746344141097054, 0...

[2023-09-11T12:54:54.496Z] 2         2     2.0       2  [0.027977732393324833, 0.15624415......, kwargs: {'timeout': 120} (api_request.py:62)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:10 - DEBUG - ci_test]: (api_response) : (insert count: 3000, delete count: 0, upsert count: 0, timestamp: 444185887582715905, success count: 3000, err count: 0)  (api_request.py:37)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:18:16 - INFO - ci_test]: index info: [] (test_action_second_deployment.py:58)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:20:27 - ERROR - pymilvus.decorators]: RPC error: [describe_collection], <DescribeCollectionException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:348 github.com/milvus-io/milvus/internal/proxy.(*ddTaskQueue).Enqueue

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:637 github.com/milvus-io/milvus/internal/proxy.(*Proxy).DescribeCollection: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-11 12:20:17.159496', 'RPC error': '2023-09-11 12:20:27.635698'}> (decorators.py:126)

[2023-09-11T12:54:54.496Z] [2023-09-11 12:20:27 - ERROR - pymilvus.decorators]: RPC error: [create_index], <DescribeCollectionException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:408 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:422 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:348 github.com/milvus-io/milvus/internal/proxy.(*ddTaskQueue).Enqueue

[2023-09-11T12:54:54.496Z] /go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:637 github.com/milvus-io/milvus/internal/proxy.(*Proxy).DescribeCollection: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2023-09-11 12:20:17.159375', 'RPC error': '2023-09-11 12:20:27.635931'}>

Expected Behavior
No response
Steps To Reproduce
No response
Milvus Log
failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_for_release_cron/detail/deploy_test_for_release_cron/889/pipeline
log:
artifacts-rocksmq-standalone-reinstall-889-pytest-logs.tar.gz
artifacts-rocksmq-standalone-reinstall-889-server-logs.tar.gz
Anything else?
No response

Duplicate issue #27089: [Bug]: [perf-nightly] Milvus perform concurrent query raise errors"context deadline exceeded: unrecoverable error" with hnsw index gist1m 768d data
Created: 2023-09-14T06:35:12Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version:2.3.0-20230913-6fcd3e8f
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior
server:
NAME                                                              READY   STATUS      RESTARTS         AGE     IP              NODE         NOMINATED NODE   READINESS GATES
perf-query-test15400-7-85-3539-etcd-0                             1/1     Running     0                56m     10.104.12.75    4am-node17   <none>           <none>
perf-query-test15400-7-85-3539-milvus-standalone-8c49587585n9s9   1/1     Running     0                56m     10.104.21.33    4am-node24   <none>           <none>
perf-query-test15400-7-85-3539-minio-58fdf6df7b-zgkg6             1/1     Running     0                56m     10.104.12.73    4am-node17   <none>           <none> (cli_client.py:132)

client log:
[2023-09-13 14:49:24,769 - ERROR - fouram]: RPC error: [query], <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:484 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:498 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2689 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Query
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:838 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Query: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unrecoverable error)>, <Time:{'RPC start': '2023-09-13 14:49:11.582082', 'RPC error': '2023-09-13 14:49:24.768951'}> (decorators.py:108)
[2023-09-13 14:49:24,771 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:484 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:498 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2689 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Query
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:838 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Query: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unrecoverable error)> (api_request.py:53)
[2023-09-13 14:49:24,771 - ERROR - fouram]: [CheckFunc] query request check failed, response:<MilvusException: (code=1, message=syncTimestamp Failed:stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:484 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
/go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:498 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:120 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
/go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:321 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).AllocTimestamp
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:61 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).alloc
/go/src/github.com/milvus-io/milvus/internal/proxy/timestamp.go:83 github.com/milvus-io/milvus/internal/proxy.(*timestampAllocator).AllocOne
/go/src/github.com/milvus-io/milvus/internal/proxy/task_scheduler.go:170 github.com/milvus-io/milvus/internal/proxy.(*baseTaskQueue).Enqueue
/go/src/github.com/milvus-io/milvus/internal/proxy/impl.go:2689 github.com/milvus-io/milvus/internal/proxy.(*Proxy).Query
/go/src/github.com/milvus-io/milvus/internal/distributed/proxy/service.go:838 github.com/milvus-io/milvus/internal/distributed/proxy.(*Server).Query: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unrecoverable error)> (func_check.py:46)

Expected Behavior
No response
Steps To Reproduce
1. create a collection 
        2. build hnsw index on vector column
        3. insert 1m gist of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. count the total number of rows
        7. load collection
        8. perform concurrent query-> raise error
Milvus Log
No response
Anything else?
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'dim': 768,
                                                    'dataset_name': 'gist',
                                                    'dataset_size': '1m',
                                                    'ni_per': 5000},
                                 'load_params': {},
                                 'search_params': {},
                                 'resource_groups_params': {'reset': False},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 8,
                                                                  'efConstruction': 200}},
                                 'concurrent_params': {'concurrent_number': 1,
                                                       'during_time': 600,
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'query',
                                                       'weight': 1,
                                                       'params': {'expr': 'id '
                                                                          'in '
                                                                          '[1, '
                                                                          '100, '
                                                                          '1000]'}}]},

Closing comment (2023-09-19T03:53:17Z): the same as #27012