Repository: milvus-io/milvus

Issue #23966: [Bug]: [benchmark] diskann inserted 100 million data, load failed, and reported "collection xxx has not been loaded to memory or load failed"
Created: 2023-05-09T03:41:25Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version: master-20230506-7f5294b1
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    pulsar
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0.dev12
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior
The memory resources are sufficient, but the load fails and "collection xxx has not been loaded to memory or load failed" is reported.
argo task : fouramf-shtt7-kg49g
resource :
    dataNode:
      replicas: 1
      resources:
        requests:
            memory: 64Gi
            cpu: 8.0
        limits:
            memory: 64Gi
            cpu: 8.0
    indexNode:
      replicas: 1
      resources:
        requests:
            memory: 64Gi
            cpu: 8.0
        limits:
            memory: 64Gi
            cpu: 8.0
            ephemeral-storage: 256Gi
    queryNode:
      replicas: 1
      resources:
        requests:
          memory: 64Gi
          cpu: 8.0
        limits:
          memory: 64Gi
          cpu: 8.0
          ephemeral-storage: 256Gi

server:
fouramf-shtt7-kg49g-97-6367-etcd-0                                1/1     Running     0               4m18s   10.104.24.172   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-etcd-1                                1/1     Running     0               4m18s   10.104.22.9     4am-node26   <none>           <none>
fouramf-shtt7-kg49g-97-6367-etcd-2                                1/1     Running     0               4m17s   10.104.6.153    4am-node13   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-datacoord-5bbcb78b65-kprns     1/1     Running     0               4m18s   10.104.24.167   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-datanode-55d494bd48-79tc2      1/1     Running     0               4m18s   10.104.23.92    4am-node27   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-indexcoord-67d56f59cd-fgmlg    1/1     Running     0               4m18s   10.104.19.67    4am-node28   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-indexnode-6cffd4d966-vq9jb     1/1     Running     0               4m18s   10.104.24.169   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-proxy-556c76586-qfd9h          1/1     Running     0               4m18s   10.104.24.170   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-querycoord-75497595df-7gvcl    1/1     Running     0               4m18s   10.104.24.168   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-querynode-7678576b76-xm7nz     1/1     Running     0               4m18s   10.104.19.66    4am-node28   <none>           <none>
fouramf-shtt7-kg49g-97-6367-milvus-rootcoord-746c94497c-26wrl     1/1     Running     0               4m18s   10.104.19.68    4am-node28   <none>           <none>
fouramf-shtt7-kg49g-97-6367-minio-0                               1/1     Running     0               4m17s   10.104.6.152    4am-node13   <none>           <none>
fouramf-shtt7-kg49g-97-6367-minio-1                               1/1     Running     0               4m17s   10.104.22.12    4am-node26   <none>           <none>
fouramf-shtt7-kg49g-97-6367-minio-2                               1/1     Running     0               4m17s   10.104.5.179    4am-node12   <none>           <none>
fouramf-shtt7-kg49g-97-6367-minio-3                               1/1     Running     0               4m17s   10.104.20.185   4am-node22   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-bookie-0                       1/1     Running     0               4m18s   10.104.24.174   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-bookie-1                       1/1     Running     0               4m17s   10.104.22.14    4am-node26   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-bookie-2                       1/1     Running     0               4m17s   10.104.6.159    4am-node13   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-bookie-init-s7gjq              0/1     Completed   0               4m18s   10.104.24.162   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-broker-0                       1/1     Running     0               4m18s   10.104.24.160   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-proxy-0                        1/1     Running     0               4m18s   10.104.19.69    4am-node28   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-pulsar-init-6ztch              0/1     Completed   0               4m18s   10.104.24.161   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-recovery-0                     1/1     Running     0               4m18s   10.104.22.7     4am-node26   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-zookeeper-0                    1/1     Running     0               4m18s   10.104.24.173   4am-node29   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-zookeeper-1                    1/1     Running     0               3m32s   10.104.5.181    4am-node12   <none>           <none>
fouramf-shtt7-kg49g-97-6367-pulsar-zookeeper-2                    1/1     Running     0               2m57s   10.104.22.16    4am-node26   <none>           <none> 

client log:
[2023-05-08 05:24:05,050 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_xBvJKZYG): 99800000 (base.py:318)
[2023-05-08 05:24:05,239 -  INFO - fouram]: [Base] Start inserting, ids: 99850000 - 99899999, data size: 100,000,000 (base.py:164)
[2023-05-08 05:24:07,001 -  INFO - fouram]: [Time] Collection.insert run in 1.7616s (api_request.py:41)
[2023-05-08 05:24:07,004 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_xBvJKZYG): 99800000 (base.py:318)
[2023-05-08 05:24:07,942 -  INFO - fouram]: [Base] Start inserting, ids: 99900000 - 99949999, data size: 100,000,000 (base.py:164)
[2023-05-08 05:24:09,767 -  INFO - fouram]: [Time] Collection.insert run in 1.8244s (api_request.py:41)
[2023-05-08 05:24:09,770 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_xBvJKZYG): 99900000 (base.py:318)
[2023-05-08 05:24:09,937 -  INFO - fouram]: [Base] Start inserting, ids: 99950000 - 99999999, data size: 100,000,000 (base.py:164)
[2023-05-08 05:24:11,600 -  INFO - fouram]: [Time] Collection.insert run in 1.6626s (api_request.py:41)
[2023-05-08 05:24:11,604 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_xBvJKZYG): 99900000 (base.py:318)
[2023-05-08 05:24:11,691 -  INFO - fouram]: [Base] Total time of insert: 3135.3359s, average number of vector bars inserted per second: 31894.5093, average time to insert 50000 vectors per time: 1.5677s (base.py:235)
[2023-05-08 05:24:11,692 -  INFO - fouram]: [Base] Start flush collection fouram_xBvJKZYG (base.py:133)
[2023-05-08 05:24:14,714 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_xBvJKZYG): 100000000 (base.py:318)
[2023-05-08 05:24:14,719 -  INFO - fouram]: [Base] Params of index: {'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}} (base.py:296)
[2023-05-08 05:24:14,719 -  INFO - fouram]: [Base] Start build index of DISKANN for collection fouram_xBvJKZYG, params:{'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}} (base.py:283)
[2023-05-08 23:00:39,196 -  INFO - fouram]: [Time] Index run in 63384.4759s (api_request.py:41)
[2023-05-08 23:00:39,197 -  INFO - fouram]: [CommonCases] RT of build index DISKANN: 63384.4759s (common_cases.py:87)
[2023-05-08 23:00:39,200 -  INFO - fouram]: [Base] Params of index: {'index_type': 'DISKANN', 'metric_type': 'L2', 'params': {}} (base.py:296)
[2023-05-08 23:00:39,200 -  INFO - fouram]: [CommonCases] Prepare index DISKANN done. (common_cases.py:90)
[2023-05-08 23:00:39,200 -  INFO - fouram]: [CommonCases] No scalars need to be indexed. (common_cases.py:95)
[2023-05-08 23:00:39,200 -  INFO - fouram]: [Base] Start load collection fouram_xBvJKZYG,replica_number:1,kwargs:{} (base.py:139)
[2023-05-08 23:10:41,078 - ERROR - fouram]: RPC error: [get_loading_progress], <MilvusException: (code=1, message=collection 441324366066876535 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-05-08 23:10:41.076732', 'RPC error': '2023-05-08 23:10:41.078363'}> (decorators.py:108)
[2023-05-08 23:10:41,080 - ERROR - fouram]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=collection 441324366066876535 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-05-08 23:00:39.217361', 'RPC error': '2023-05-08 23:10:41.080019'}> (decorators.py:108)
[2023-05-08 23:10:41,080 - ERROR - fouram]: RPC error: [load_collection], <MilvusException: (code=1, message=collection 441324366066876535 has not been loaded to memory or load failed)>, <Time:{'RPC start': '2023-05-08 23:00:39.200687', 'RPC error': '2023-05-08 23:10:41.080147'}> (decorators.py:108)
[2023-05-08 23:10:41,082 - ERROR - fouram]: (api_response) : <MilvusException: (code=1, message=collection 441324366066876535 has not been loaded to memory or load failed)> (api_request.py:49)
[2023-05-08 23:10:41,082 - ERROR - fouram]: [CheckFunc] load request check failed, response:<MilvusException: (code=1, message=collection 441324366066876535 has not been loaded to memory or load failed)> (func_check.py:52)

memory usage:

Expected Behavior
No response
Steps To Reproduce
1. create a collection or use an existing collection
        2. build index on vector column
        3. insert a certain number of vectors
        4. flush collection
        5. build index on vector column with the same parameters
        6. build index on on scalars column or not
        7. count the total number of rows
        8. load collection  ==> failed
        9. perform concurrent operations
        10. clean all collections or not
Milvus Log
No response
Anything else?
No response

Duplicate issue #24903: [Bug]: [Nightly]Plenty of cases failed for loading collection timeout and extend 2 more hours
Created: 2023-06-15T02:51:03Z
Is there an existing issue for this?

 I have searched the existing issues

Environment
- Milvus version: 56380ea
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka): kafka
- SDK version(e.g. pymilvus v2.0.0rc2): 2.4.0.dev67
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:
Current Behavior
Plenty of cases failed for loading collection timeout and extend 2 more hours.
seems something wrong with diskann. https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/master/403/pipeline/267
[2023-06-14T23:25:37.891Z] [2023-06-14 19:54:57 - ERROR - pymilvus.decorators]: RPC error: [wait_for_loading_collection], <MilvusException: (code=1, message=wait for loading collection timeout, collection: search_collection_3AVMcsQi)>, <Time:{'RPC start': '2023-06-14 19:52:57.103531', 'RPC error': '2023-06-14 19:54:57.446694'}> (decorators.py:108)
[2023-06-14T23:25:37.891Z] [2023-06-14 19:54:57 - WARNING - pymilvus.decorators]: �[93mRetry timeout: 120s�[0m (decorators.py:79)
[2023-06-14T23:25:37.891Z] [2023-06-14 19:54:57 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=1, message=Retry timeout: 120s, message=wait for loading collection timeout, collection: search_collection_3AVMcsQi)>, <Time:{'RPC start': '2023-06-14 19:52:56.842257', 'RPC error': '2023-06-14 19:54:57.447009'}> (decorators.py:108)
[2023-06-14T23:25:37.891Z] [2023-06-14 19:54:57 - ERROR - ci_test]: Traceback (most recent call last):
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-06-14T23:25:37.891Z]     return func(self, *args, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 791, in load_collection
[2023-06-14T23:25:37.891Z]     self.wait_for_loading_collection(collection_name, timeout)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-06-14T23:25:37.891Z]     raise e
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-06-14T23:25:37.891Z]     return func(*args, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-06-14T23:25:37.891Z]     ret = func(self, *args, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 85, in handler
[2023-06-14T23:25:37.891Z]     raise e
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 50, in handler
[2023-06-14T23:25:37.891Z]     return func(self, *args, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 813, in wait_for_loading_collection
[2023-06-14T23:25:37.891Z]     raise MilvusException(message=f"wait for loading collection timeout, collection: {collection_name}")
[2023-06-14T23:25:37.891Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=wait for loading collection timeout, collection: search_collection_3AVMcsQi)>
[2023-06-14T23:25:37.891Z] 
[2023-06-14T23:25:37.891Z] The above exception was the direct cause of the following exception:
[2023-06-14T23:25:37.891Z] 
[2023-06-14T23:25:37.891Z] Traceback (most recent call last):
[2023-06-14T23:25:37.891Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 26, in inner_wrapper
[2023-06-14T23:25:37.891Z]     res = func(*args, **_kwargs)
[2023-06-14T23:25:37.891Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 57, in api_request
[2023-06-14T23:25:37.891Z]     return func(*arg, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 375, in load
[2023-06-14T23:25:37.891Z]     conn.load_collection(self._name, replica_number=replica_number, timeout=timeout, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 109, in handler
[2023-06-14T23:25:37.891Z]     raise e
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 105, in handler
[2023-06-14T23:25:37.891Z]     return func(*args, **kwargs)
[2023-06-14T23:25:37.891Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 136, in handler
[2023-06-14T23:25:37.891Z]     ret = func(self, *args, **kwargs)
[2023-06-14T23:25:37.892Z]   File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 80, in handler
[2023-06-14T23:25:37.892Z]     raise MilvusException(e.code, f"{timeout_msg}, message={e.message}") from e
[2023-06-14T23:25:37.892Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=Retry timeout: 120s, message=wait for loading collection timeout, collection: search_collection_3AVMcsQi)>
[2023-06-14T23:25:37.892Z]  (api_request.py:39)
[2023-06-14T23:25:37.892Z] [2023-06-14 19:54:57 - ERROR - ci_test]: (api_response) : <MilvusException: (code=1, message=Retry timeout: 120s, message=wait for loading collection timeout, collection: search_collection_3AVMcsQi)> (api_request.py:40)

Expected Behavior
pass
Steps To Reproduce
No response
Milvus Log
artifacts-milvus-distributed-kafka-nightly-403-pymilvus-e2e-logs.tar.gz
Anything else?
No response

Closing comment (2023-06-15T04:00:26Z): Same as #23966, knowhere behavior changed
/assign @cqy123456 is fixing this