Repository: milvus-io/milvus

Issue #23262: [Bug]: query node oom killed due to inaccurate memory prediction based on container limit
Created: 2023-04-07T02:13:20Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version: 2.2.5
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka): Kafka   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others: 
- image: v2.2.0-20230402-6d5ec9c-28175ab

Current Behavior
set pod request memory with 2g and limit 9g
query node crashed and restart
client sdk got load failed response and raise an error
Expected Behavior
query node keeps stable and report failure right away when memory is not sufficient
Steps To Reproduce
create a collection
build autoindex index
insert 25m data
load collection and error raised after a few minutes

Milvus Log
[2023-04-02 18:33:44,908] [   DEBUG] - Start build index for last file (milvus_benchmark.runners.locust:318)
2[2023-04-02 18:33:44,908] [    INFO] - Building index start, collection_name: sift_25m_128_l2, index_type: AUTOINDEX, metric_type: L2 (milvus_benchmark.client:302)
3[2023-04-02 18:33:44,908] [   DEBUG] - collection: sift_25m_128_l2 Index params: {'index_type': 'AUTOINDEX', 'metric_type': 'L2', 'params': {}} (milvus_benchmark.client:311)
4[2023-04-02 19:06:06,741] [   DEBUG] - Building index done, collection_name: sift_25m_128_l2, response: Status(code=0, message=) (milvus_benchmark.client:313)
5[2023-04-02 19:06:06,744] [   DEBUG] - Milvus create_index run in 1941.8366s (milvus_benchmark.client:57)
6[2023-04-02 19:06:06,744] [   DEBUG] - {'flush_time': 2.0255, 'build_time': 1941.8366} (milvus_benchmark.runners.locust:322)
7[2023-04-02 19:06:06,746] [   DEBUG] - Row count: 25000000 in collection: <sift_25m_128_l2> (milvus_benchmark.client:528)
8[2023-04-02 19:06:06,746] [    INFO] - 25000000 (milvus_benchmark.runners.locust:323)
9[2023-04-02 19:06:06,747] [    INFO] - Start load collection (milvus_benchmark.runners.locust:324)
10[2023-04-02 19:08:40,487] [ WARNING] - �[93m[get_loading_progress] retry:4, cost: 0.27s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, unavailable>�[0m (pymilvus.decorators:71)
11[2023-04-02 19:08:40,763] [ WARNING] - �[93m[get_loading_progress] retry:5, cost: 0.81s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, unavailable>�[0m (pymilvus.decorators:71)
12[2023-04-02 19:08:41,576] [ WARNING] - �[93m[get_loading_progress] retry:6, cost: 2.43s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait>�[0m (pymilvus.decorators:71)
13[2023-04-02 19:08:44,009] [ WARNING] - �[93m[get_loading_progress] retry:7, cost: 7.29s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait>�[0m (pymilvus.decorators:71)
14[2023-04-02 19:08:51,302] [ WARNING] - �[93m[get_loading_progress] retry:8, cost: 21.87s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, internal: Milvus Proxy is not ready yet. please wait>�[0m (pymilvus.decorators:71)
15[2023-04-02 19:09:13,175] [ WARNING] - �[93m[get_loading_progress] retry:9, cost: 60.00s, reason: <_MultiThreadedRendezvous: StatusCode.UNAVAILABLE, unavailable>�[0m (pymilvus.decorators:71)
16[2023-04-02 19:10:13,182] [   ERROR] - RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 440521673462515474 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-04-02 19:08:40.226804', 'RPC error': '2023-04-02 19:10:13.182455'}> (pymilvus.decorators:108)
17[2023-04-02 19:10:13,184] [   ERROR] - RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 440521673462515474 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-04-02 19:06:06.771396', 'RPC error': '2023-04-02 19:10:13.184428'}> (pymilvus.decorators:108)
18[2023-04-02 19:10:13,184] [   ERROR] - RPC error: [load_collection], <MilvusException: (code=1, message=collection 440521673462515474 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-04-02 19:06:06.747123', 'RPC error': '2023-04-02 19:10:13.184549'}> (pymilvus.decorators:108)
19[2023-04-02 19:10:13,184] [   ERROR] - <MilvusException: (code=1, message=collection 440521673462515474 has not been loaded to memory or load failed)> (milvus_benchmark.main:120)
20[2023-04-02 19:10:13,187] [   ERROR] - Traceback (most recent call last):
Anything else?
No response

Duplicate issue #23315: [Bug]:  Milvus insert 100m data and build hnsw index, then concurrent search and query reports error "fail to <search/query> on all shard leaders"
Created: 2023-04-10T03:07:52Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version:master-20230407-dbdd879d
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior
case : test_concurrent_locust_hnsw_dql_filter_insert_cluster
argo task : platform-fouramf-wd9qx
server
qtp-3-lbmlq-etcd-0                                                1/1     Running     0               4m1s    10.104.9.69     4am-node14   <none>           <none>
qtp-3-lbmlq-etcd-1                                                1/1     Running     0               4m1s    10.104.6.190    4am-node13   <none>           <none>
qtp-3-lbmlq-etcd-2                                                1/1     Running     0               4m1s    10.104.1.132    4am-node10   <none>           <none>
qtp-3-lbmlq-milvus-datacoord-594bf589c5-nt8mm                     1/1     Running     0               4m1s    10.104.9.59     4am-node14   <none>           <none>
qtp-3-lbmlq-milvus-datanode-7cbc5b998-845rq                       1/1     Running     0               4m1s    10.104.1.125    4am-node10   <none>           <none>
qtp-3-lbmlq-milvus-indexcoord-746db8996-qxfzb                     1/1     Running     0               4m1s    10.104.6.188    4am-node13   <none>           <none>
qtp-3-lbmlq-milvus-indexnode-7fc4b46fc-xnk6r                      1/1     Running     0               4m1s    10.104.4.48     4am-node11   <none>           <none>
qtp-3-lbmlq-milvus-proxy-759bc87988-ngzjh                         1/1     Running     0               4m1s    10.104.9.62     4am-node14   <none>           <none>
qtp-3-lbmlq-milvus-querycoord-5544c4d695-69n9s                    1/1     Running     0               4m1s    10.104.6.186    4am-node13   <none>           <none>
qtp-3-lbmlq-milvus-querynode-88796fbc5-gvfgk                      1/1     Running     0               4m1s    10.104.6.185    4am-node13   <none>           <none>
qtp-3-lbmlq-milvus-rootcoord-8675dd5ff9-dtdf2                     1/1     Running     0               4m1s    10.104.9.58     4am-node14   <none>           <none>
qtp-3-lbmlq-minio-0                                               1/1     Running     0               4m1s    10.104.9.71     4am-node14   <none>           <none>
qtp-3-lbmlq-minio-1                                               1/1     Running     0               4m1s    10.104.1.129    4am-node10   <none>           <none>
qtp-3-lbmlq-minio-2                                               1/1     Running     0               4m1s    10.104.6.193    4am-node13   <none>           <none>
qtp-3-lbmlq-minio-3                                               1/1     Running     0               4m      10.104.4.50     4am-node11   <none>           <none>
qtp-3-lbmlq-pulsar-bookie-0                                       1/1     Running     0               4m1s    10.104.9.72     4am-node14   <none>           <none>
qtp-3-lbmlq-pulsar-bookie-1                                       1/1     Running     0               4m1s    10.104.1.133    4am-node10   <none>           <none>
qtp-3-lbmlq-pulsar-bookie-2                                       1/1     Running     0               4m      10.104.6.195    4am-node13   <none>           <none>
qtp-3-lbmlq-pulsar-bookie-init-997xq                              0/1     Completed   0               4m1s    10.104.9.57     4am-node14   <none>           <none>
qtp-3-lbmlq-pulsar-broker-0                                       1/1     Running     0               4m1s    10.104.9.60     4am-node14   <none>           <none>
qtp-3-lbmlq-pulsar-proxy-0                                        1/1     Running     0               4m1s    10.104.9.63     4am-node14   <none>           <none>
qtp-3-lbmlq-pulsar-pulsar-init-n62d7                              0/1     Completed   0               4m1s    10.104.6.187    4am-node13   <none>           <none>
qtp-3-lbmlq-pulsar-recovery-0                                     1/1     Running     0               4m1s    10.104.1.126    4am-node10   <none>           <none>
qtp-3-lbmlq-pulsar-zookeeper-0                                    1/1     Running     0               4m1s    10.104.9.68     4am-node14   <none>           <none>
qtp-3-lbmlq-pulsar-zookeeper-1                                    1/1     Running     0               3m19s   10.104.1.136    4am-node10   <none>           <none>
qtp-3-lbmlq-pulsar-zookeeper-2                                    1/1     Running     0               2m44s   10.104.6.197    4am-node13   <none>           <none>

client error log:
fouram_log (1).err.zip
Expected Behavior
No response
Steps To Reproduce
1. create a collection or use an existing collection
        2. build index on vector column
        3. insert a certain number of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. build index on on scalars column or not
        7. count the total number of rows
        8. load collection
        9. perform concurrent operations （query 、search 、ddl）
        10. clean all collections or not
Milvus Log
No response
Anything else?
No response

Closing comment (2023-04-10T12:34:55Z): this is related to #23262