Repository: regro/regolith

Issue:
helper: general lister
UC1:
1. simon types ``regolith helper list people``
1. lister lists the ids of all the people in people

UC2:
1. S types ``rh list people --kv_filter name simon``
1. l lists the id of all people in people who's name contains simon

UC3:
1. S types ``rh list people --kv_filter name simon -r name position``
1. l lists the id, name and position of all people in people who's name contains simon

UC4:
1. S types ``rh list people -k``
1. l returns a list of the available keys and their description (if available) in collection people

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=h1) Report
> Merging [#475](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=desc) into [master](https://codecov.io/gh/regro/regolith/commit/bf28e43ecac7e5469af4d3ab2efc25937db65034&el=desc) will **increase** coverage by `0.24%`.
> The diff coverage is `92.85%`.

[![Impacted file tree graph](https://codecov.io/gh/regro/regolith/pull/475/graphs/tree.svg?width=650&height=150&src=pr&token=YOWfAqEPVS)](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #475      +/-   ##
==========================================
+ Coverage   64.10%   64.34%   +0.24%     
==========================================
  Files          54       55       +1     
  Lines        4948     4990      +42     
==========================================
+ Hits         3172     3211      +39     
- Misses       1776     1779       +3     
```


| [Impacted Files](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [regolith/helpers/l\_generalhelper.py](https://codecov.io/gh/regro/regolith/pull/475/diff?src=pr&el=tree#diff-cmVnb2xpdGgvaGVscGVycy9sX2dlbmVyYWxoZWxwZXIucHk=) | `92.68% <92.68%> (ø)` | |
| [regolith/helper.py](https://codecov.io/gh/regro/regolith/pull/475/diff?src=pr&el=tree#diff-cmVnb2xpdGgvaGVscGVyLnB5) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=footer). Last update [bf28e43...2186a09](https://codecov.io/gh/regro/regolith/pull/475?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

dy2403: Hi @sbillinge!
I have a question about the -k and exactly it should retrieve. (From UC4)
Are keys like name, position, etc.? Should it be retrieved from schemas?
Or should I establish the keys available for each collection in the function?

sbillinge: > Hi @sbillinge!
> I have a question about the -k and exactly it should retrieve. (From UC4)
> Are keys like name, position, etc.? Should it be retrieved from schemas?
> Or should I establish the keys available for each collection in the function?

-k there stands for --keys and it lists all the keys that I can give to the lister either to search over, or to return.  Otherwise I may not remember what I have available to search over.
dy2403: @sbillinge ready for review and merge!
dy2403: I do have a prum! Should I finish it when it is merged, or when we guarantee that it is working when we test it?
dy2403: Hi prof. @sbillinge,
I think this is ready to review and merge!

Patch:
diff --git a/news/generallister.rst b/news/generallister.rst
new file mode 100644
index 00000000..5847f0f5
--- /dev/null
+++ b/news/generallister.rst
@@ -0,0 +1,12 @@
+**Added:**
+ * a helper for listing filtered data from collections in the database.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None
\ No newline at end of file
diff --git a/regolith/helper.py b/regolith/helper.py
index ca11f826..49c54542 100644
--- a/regolith/helper.py
+++ b/regolith/helper.py
@@ -16,6 +16,7 @@ from regolith.helpers import u_milestonehelper as u_milestone
 from regolith.helpers import u_contacthelper as u_contact
 from regolith.helpers import l_todohelper as l_todo
 from regolith.helpers import u_finishprumhelper as u_finishprum
+from regolith.helpers import l_generalhelper as l_general
 
 HELPERS = {
     "hello": (hello.HelloHelper, hello.subparser),
@@ -34,6 +35,7 @@ HELPERS = {
     "u_milestone": (u_milestone.MilestoneUpdaterHelper, u_milestone.subparser),
     "l_todo": (l_todo.TodoListerHelper, l_todo.subparser),
     "finish_prum": (u_finishprum.FinishprumUpdaterHelper, u_finishprum.subparser),
+    "lister": (l_general.GeneralListerHelper, l_general.subparser),
 }
 
 
diff --git a/regolith/helpers/l_generalhelper.py b/regolith/helpers/l_generalhelper.py
new file mode 100644
index 00000000..21ebc971
--- /dev/null
+++ b/regolith/helpers/l_generalhelper.py
@@ -0,0 +1,100 @@
+"""Helper for listing filtered data from collections in the database.
+"""
+
+from regolith.helpers.basehelper import SoutHelperBase
+from regolith.fsclient import _id_key
+from regolith.tools import (
+    all_docs_from_collection,
+    get_pi_id,
+    search_collection,
+    collection_str,
+)
+HELPER_TARGET = "lister"
+
+def subparser(subpi):
+    subpi.add_argument(
+        "coll",
+        help='Collection that the lister is going to list from.')
+    subpi.add_argument(
+        "-f","--kv_filter",
+        nargs="+",
+        help="Search the given collection by key-value pairs. "
+             "e.g. 'regolith helper lister -f name simon' will "
+             "return the id of all the people who's name contains simon.")
+    subpi.add_argument(
+        "-r",  "--return_fields",
+        nargs="+",
+        help="Specify from which keys to print values. "
+             "e.g. 'regolith helper lister people -r name status title' "
+             "will return the name, status, and title of all the people "
+             "in the collection. If no argument is given the default is just the id.")
+    subpi.add_argument(
+        "-k", "--keys",
+        action="store_true",
+        help="List of the available keys in the collection.")
+    return subpi
+
+class GeneralListerHelper(SoutHelperBase):
+    """Helper for listing filtered data from collections in the database.
+    """
+    # btype must be the same as helper target in helper.py
+    btype = HELPER_TARGET
+
+    def construct_global_ctx(self):
+        """Constructs the global context"""
+        super().construct_global_ctx()
+        gtx = self.gtx
+        rc = self.rc
+        needed_dbs = [rc.coll]
+        if "groups" in needed_dbs:
+            rc.pi_id = get_pi_id(rc)
+        try:
+            if not rc.database:
+                rc.database = rc.databases[0]["name"]
+        except BaseException:
+            pass
+        colls = [
+            sorted(
+                all_docs_from_collection(rc.client, collname), key=_id_key
+            )
+            for collname in needed_dbs
+        ]
+        for db, coll in zip(needed_dbs, colls):
+            gtx[db] = coll
+        gtx["all_docs_from_collection"] = all_docs_from_collection
+        gtx["float"] = float
+        gtx["str"] = str
+        gtx["zip"] = zip
+
+    def sout(self):
+        rc = self.rc
+        coll = self.gtx[rc.coll]
+        keys = sorted(set([i for k in coll for i in k.keys()]))
+        list_keys = []
+        if rc.kv_filter:
+            list_keys.extend([rc.kv_filter[n] for n in range(len(rc.kv_filter)) if n % 2 == 0])
+        if rc.return_fields:
+            list_keys.extend(rc.return_fields)
+        for i in list_keys:
+            if i not in keys:
+                raise ValueError(f"{i} is not a valid key. Please choose a valid key from: {keys}")
+        if len(coll) == 0:
+            raise RuntimeError('This collection is empty or does not exist. Please inform a valid collection.')
+        if rc.kv_filter:
+            filter_result = search_collection(coll, rc.kv_filter).strip()
+            if filter_result == '':
+                print("There are no results that match your search.")
+            else:
+                if rc.return_fields:
+                    print("Results of your search:\n"
+                            f"{(search_collection(coll, rc.kv_filter, rc.return_fields).strip())}")
+                else:
+                    print(f"Results of your search:\n{filter_result}")
+        if rc.return_fields and not rc.kv_filter:
+            print(f"Results of your search:\n{collection_str(coll, rc.return_fields).strip()}")
+        if rc.keys:
+            print(f"Available keys:\n{keys}")
+            return
+        if not rc.kv_filter and not rc.return_fields:
+            print(f"Results of your search:\n{collection_str(coll).strip()}")
+        return
diff --git a/regolith/schemas.py b/regolith/schemas.py
index a38ad6bb..0a537df7 100644
--- a/regolith/schemas.py
+++ b/regolith/schemas.py
@@ -1468,7 +1468,7 @@ SCHEMAS = {
             "type": "string",
         },
         "institution": {
-            "description": "name of the inistitution",
+            "description": "name of the institution",
             "required": True,
             "type": "string",
         },


Test patch:
diff --git a/tests/test_helpers.py b/tests/test_helpers.py
index 8a86bcd3..8b0a8b25 100644
--- a/tests/test_helpers.py
+++ b/tests/test_helpers.py
@@ -159,6 +159,36 @@ helper_map = [
      "20sb_firstprojectum     status:finished\n"
      "Please rerun the helper specifying the complete ID.\n"
      ),
+
+    #UC1
+    (["helper", "lister", "people"],
+     "Results of your search:\nsbillinge    \nscopatz\n"),
+    (["helper", "lister", "people", "--kv_filter", "name", "simon"],
+     "Results of your search:\n"
+     "sbillinge\n"),
+    (["helper", "lister", "people", "--kv_filter", "name", "simon", "--return_fields", "name", "position"],
+     "Results of your search:\nsbillinge    name: Simon J. L. Billinge    position: professor\n"),
+    (["helper", "lister", "people", "--keys"],
+     "Available keys:\n"
+     "['_id', 'activities', 'aka', 'appointments', 'avatar', 'bio', 'bios', "
+     "'committees', 'education', 'email', 'employment', 'facilities', 'funding', "
+     "'github_id', 'google_scholar_url', 'hindex', 'home_address', 'initials', "
+     "'membership', 'miscellaneous', 'name', 'office', 'orcid_id', 'position', "
+     "'publicity', 'research_focus_areas', 'service', 'skills', 'teaching', "
+     "'title', 'todos']\n"),
+    (["helper", "lister", "people", "--kv_filter", "name", "simon", "--keys"],
+     "Results of your search:\nsbillinge\n"
+     "Available keys:\n"
+     "['_id', 'activities', 'aka', 'appointments', 'avatar', 'bio', 'bios', "
+     "'committees', 'education', 'email', 'employment', 'facilities', 'funding', "
+     "'github_id', 'google_scholar_url', 'hindex', 'home_address', 'initials', "
+     "'membership', 'miscellaneous', 'name', 'office', 'orcid_id', 'position', "
+     "'publicity', 'research_focus_areas', 'service', 'skills', 'teaching', "
+     "'title', 'todos']\n"
+     ),
+    (["helper", "lister", "people", "--kv_filter", "name", "simon", "position", "singer"],
+     "There are no results that match your search.\n"
+     ),
 ]