Repository: StackStorm/st2

Issue:
sensor_service.list_value() doesn't return all the keys macthing the prefix
## SUMMARY

Inside a PollingSensor, I am trying to fetch all the key from a datastore which starts with a prefix 'entity_' like this:

`self.sensor_service.list_value(prefix="entity_")`

However, if the datastore contains several (or at least more than 100, that I could test) the method doesn't return all the keys.
Is there a pagination here? If yes, how can we get the entire list in one go? Or at least fetch the next set of keys?

### STACKSTORM VERSION

st2 3.3.0, on Python 3.6.9

##### OS, environment, install method

Post what OS you are running this on, along with any other relevant information/

Linux

## Steps to reproduce the problem

Use any sensor, store 100+ keys with the same prefix in the datastore. 
Next, try to fetch all the keys stored with the same prefix. 

## Expected Results

I expect to get a list of all the keys that have the common prefix or have a way to get the entire list.


## Actual Results

The code mentioned didn't return all the keys that were expected.


Thanks!



Discussion:
Ran into the same issue and had to implement a separate call to the API endpoint to get around it.  The sensor code uses the ST2 client to get the keys and the ST2 client supports an option to request all the key values (-1) but this is not exposed to the method in the sensor code.
Duplicate of #5097 

Can you share more details on the workaround? What is the separate call to API endpoint? 
I cannot share the actual code as it was part of a custom sensor we wrote for an internal project, but we basically used Requests to hit the ST2 API endpoint to get the keys (exactly the same way the st2 client gets this data):

https://api.stackstorm.com/api/v1/keys/#/key_value_pair_controller.get_all

With the API endpoint, you can pass the "limit=-1" parameter in the URL to have the API server pass all the values from the keystore and overcome the API server limit set to 100.

The test URL we have in Postman that would work like you need would be as follows:

https://ST2-API-ENDPOINT-URL/api/v1/keys?prefix=entity_&limit=-1



That's great! Thank you :)

Resolving patch:
diff --git a/st2client/st2client/models/core.py b/st2client/st2client/models/core.py
--- a/st2client/st2client/models/core.py
+++ b/st2client/st2client/models/core.py
@@ -180,6 +180,7 @@ def get_all(self, **kwargs):
         pack = kwargs.pop("pack", None)
         prefix = kwargs.pop("prefix", None)
         user = kwargs.pop("user", None)
+        offset = kwargs.pop("offset", 0)
 
         params = kwargs.pop("params", {})
 
@@ -195,6 +196,9 @@ def get_all(self, **kwargs):
         if user:
             params["user"] = user
 
+        if offset:
+            params["offset"] = offset
+
         response = self.client.get(url=url, params=params, **kwargs)
         if response.status_code != http_client.OK:
             self.handle_error(response)
diff --git a/st2common/st2common/services/datastore.py b/st2common/st2common/services/datastore.py
--- a/st2common/st2common/services/datastore.py
+++ b/st2common/st2common/services/datastore.py
@@ -68,7 +68,7 @@ def get_user_info(self):
     # Methods for datastore management
     ##################################
 
-    def list_values(self, local=True, prefix=None):
+    def list_values(self, local=True, prefix=None, limit=None, offset=0):
         """
         Retrieve all the datastores items.
 
@@ -78,13 +78,20 @@ def list_values(self, local=True, prefix=None):
         :param prefix: Optional key name prefix / startswith filter.
         :type prefix: ``str``
 
+        :param limit: Number of keys to get. Defaults to the configuration set at 'api.max_page_size'.
+        :type limit: ``integer``
+
+        :param offset: Number of keys to offset. Defaults to 0.
+        :type offset: ``integer``
+
         :rtype: ``list`` of :class:`KeyValuePair`
         """
         client = self.get_api_client()
         self._logger.debug("Retrieving all the values from the datastore")
 
+        limit = limit or cfg.CONF.api.max_page_size
         key_prefix = self._get_full_key_prefix(local=local, prefix=prefix)
-        kvps = client.keys.get_all(prefix=key_prefix)
+        kvps = client.keys.get_all(prefix=key_prefix, limit=limit, offset=offset)
         return kvps
 
     def get_value(self, name, local=True, scope=SYSTEM_SCOPE, decrypt=False):


Test patch:
diff --git a/st2common/tests/unit/test_datastore.py b/st2common/tests/unit/test_datastore.py
--- a/st2common/tests/unit/test_datastore.py
+++ b/st2common/tests/unit/test_datastore.py
@@ -51,14 +51,22 @@ def test_datastore_operations_list_values(self):
         self._set_mock_api_client(mock_api_client)
 
         self._datastore_service.list_values(local=True, prefix=None)
-        mock_api_client.keys.get_all.assert_called_with(prefix="core.TestSensor:")
+        mock_api_client.keys.get_all.assert_called_with(
+            prefix="core.TestSensor:", limit=100, offset=0
+        )
         self._datastore_service.list_values(local=True, prefix="ponies")
-        mock_api_client.keys.get_all.assert_called_with(prefix="core.TestSensor:ponies")
+        mock_api_client.keys.get_all.assert_called_with(
+            prefix="core.TestSensor:ponies", limit=100, offset=0
+        )
 
         self._datastore_service.list_values(local=False, prefix=None)
-        mock_api_client.keys.get_all.assert_called_with(prefix=None)
+        mock_api_client.keys.get_all.assert_called_with(
+            prefix=None, limit=100, offset=0
+        )
         self._datastore_service.list_values(local=False, prefix="ponies")
-        mock_api_client.keys.get_all.assert_called_with(prefix="ponies")
+        mock_api_client.keys.get_all.assert_called_with(
+            prefix="ponies", limit=100, offset=0
+        )
 
         # No values in the datastore
         mock_api_client = mock.Mock()
@@ -85,6 +93,18 @@ def test_datastore_operations_list_values(self):
         self.assertEqual(len(values), 2)
         self.assertEqual(values, mock_return_value)
 
+        # Test limit
+        _ = self._datastore_service.list_values(local=True, limit=1)
+        mock_api_client.keys.get_all.assert_called_with(
+            prefix="core.TestSensor:", limit=1, offset=0
+        )
+
+        # Test offset
+        _ = self._datastore_service.list_values(local=True, offset=1)
+        mock_api_client.keys.get_all.assert_called_with(
+            prefix="core.TestSensor:", limit=100, offset=1
+        )
+
     def test_datastore_operations_get_value(self):
         mock_api_client = mock.Mock()
         kvp1 = KeyValuePair()