Repository: regro/regolith

Issue:
helper: same as u_contact but for institutions
add/update an institution entry

Discussion:
sbillinge: I gave comments, but basically the tests look good so you can start on the code.....
codecov[bot]: # [Codecov](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=h1) Report
> Merging [#495](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=desc) into [master](https://codecov.io/gh/regro/regolith/commit/a2b96553a95ce33e8e3e4f4968d3aae7214b44ee&el=desc) will **increase** coverage by `0.18%`.
> The diff coverage is `79.72%`.

[![Impacted file tree graph](https://codecov.io/gh/regro/regolith/pull/495/graphs/tree.svg?width=650&height=150&src=pr&token=YOWfAqEPVS)](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #495      +/-   ##
==========================================
+ Coverage   65.98%   66.17%   +0.18%     
==========================================
  Files          57       58       +1     
  Lines        5313     5387      +74     
==========================================
+ Hits         3506     3565      +59     
- Misses       1807     1822      +15     
```


| [Impacted Files](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [regolith/schemas.py](https://codecov.io/gh/regro/regolith/pull/495/diff?src=pr&el=tree#diff-cmVnb2xpdGgvc2NoZW1hcy5weQ==) | `95.23% <ø> (ø)` | |
| [regolith/helpers/u\_institutionshelper.py](https://codecov.io/gh/regro/regolith/pull/495/diff?src=pr&el=tree#diff-cmVnb2xpdGgvaGVscGVycy91X2luc3RpdHV0aW9uc2hlbHBlci5weQ==) | `79.45% <79.45%> (ø)` | |
| [regolith/helper.py](https://codecov.io/gh/regro/regolith/pull/495/diff?src=pr&el=tree#diff-cmVnb2xpdGgvaGVscGVyLnB5) | `100.00% <100.00%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=footer). Last update [a2b9655...89cd30b](https://codecov.io/gh/regro/regolith/pull/495?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

dy2403: Hi @sbillinge!
I changed the tests to require the user to insert the id of the university (instead of using indexes). 
Also, I added a "--new" so the user can use it when adding a new institution. This way, I thought it could make it simpler to use the helper, without indexes. What do you think about this?
Also, do you want the user to be able to add/update departments? I wasn't in the UCs, but I could add it. 
Can you take a look? 
sbillinge: > Hi @sbillinge! Can you take a look at my code?
> I changed the tests to require the user to insert the id of the university (instead of using indexes).
> Also, I added a "--new" so the user can use it when adding a new institution. This way, I thought it could make it simpler to use the helper, without indexes. What do you think about this?
> Also, do you want the user to be able to add/update departments? it wasn't in the UCs, but I could add it.

I think the index workflow is better tbh and if all the helpers work the same way it is easier for people I think.  Actually, the best workflow is that if the id exists the helper just does the updates, but if the ID doesn't exist the helper returns the list and then the user can rerun by using up-arrow and adding ``-i 5`` or whatever.  But in agile programming it is better not to tweak workflow too much until you release the code, then you will get lots of feedback on what people need and want!

Yes, we would like it if all aspects of the institution could be updated with this helper.  If it takes a long time though, don't worry too much.  this is not the most crucial of the helpers (especially since we did so much of this owrk already....)
dy2403: hi @sbillinge!
Can you please review it? 

Patch:
diff --git a/news/u_institutions.rst b/news/u_institutions.rst
new file mode 100644
index 00000000..d7c2f57f
--- /dev/null
+++ b/news/u_institutions.rst
@@ -0,0 +1,12 @@
+**Added:**
+ * a helper for adding/updating institutions to the institutions collection.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None
diff --git a/regolith/helper.py b/regolith/helper.py
index 49c54542..4229114e 100644
--- a/regolith/helper.py
+++ b/regolith/helper.py
@@ -17,6 +17,7 @@ from regolith.helpers import u_contacthelper as u_contact
 from regolith.helpers import l_todohelper as l_todo
 from regolith.helpers import u_finishprumhelper as u_finishprum
 from regolith.helpers import l_generalhelper as l_general
+from regolith.helpers import u_institutionshelper as u_institutions
 
 HELPERS = {
     "hello": (hello.HelloHelper, hello.subparser),
@@ -36,6 +37,7 @@ HELPERS = {
     "l_todo": (l_todo.TodoListerHelper, l_todo.subparser),
     "finish_prum": (u_finishprum.FinishprumUpdaterHelper, u_finishprum.subparser),
     "lister": (l_general.GeneralListerHelper, l_general.subparser),
+    "u_institutions": (u_institutions.InstitutionsUpdaterHelper, u_institutions.subparser),
 }
 
 
diff --git a/regolith/helpers/u_institutionshelper.py b/regolith/helpers/u_institutionshelper.py
new file mode 100644
index 00000000..f6317319
--- /dev/null
+++ b/regolith/helpers/u_institutionshelper.py
@@ -0,0 +1,198 @@
+"""
+Helper for updating/adding  to the projecta collection.
+"""
+from regolith.helpers.basehelper import DbHelperBase
+from regolith.fsclient import _id_key
+from regolith.tools import all_docs_from_collection, fragment_retrieval
+from itertools import chain
+import uuid
+import datetime as dt
+
+TARGET_COLL = "institutions"
+
+def subparser(subpi):
+    subpi.add_argument("institution_id",
+                       help="id of the institution, e.g., columbiau.")
+    subpi.add_argument("-n","--name",
+                        help="Full name of the institution")
+    subpi.add_argument("-i","--index",
+                       help="Index of the item in the enumerated list to update.",
+                       type=int)
+    subpi.add_argument("--city",
+                       help="The city where the institution is. "
+                            "Required for a new institution.")
+    subpi.add_argument("--country",
+                       help="The country where the institution is. "
+                            "Required for a new institution")
+    subpi.add_argument("--state",
+                       help="The state where the institution is. "
+                            "Required for a new institution if institution's country is US.")
+    subpi.add_argument("--zip",
+                       help="zipcode of the institution. "
+                            "Required for a new institution if institution's country is US.")
+    subpi.add_argument("-a", "--aka",
+                       nargs='+',
+                       help="List of all the different names this "
+                            "institution is known by.")
+    subpi.add_argument("--dept_id",
+                       help="e.g. physics.")
+    subpi.add_argument("--dept_name",
+                       help="Department canonical name, e.g., Department of Physics. "
+                            "Required if --dept_id supplied and it is a new department")
+    subpi.add_argument("--dept_aka",
+                       nargs='+',
+                       help="Department aliases, e.g., dept. of physics.")
+    subpi.add_argument("--school_id",
+                       help="Short name for the school, e.g., SEAS.")
+    subpi.add_argument("--school_name",
+                       help="Full canonical name, e.g., School of Engineering and Applied Science. "
+                            "Required if --school_id supplied and it is a new school")
+    subpi.add_argument("--school_aka",
+                       nargs='+',
+                       help="School aliases.")
+    # Do not delete --database arg
+    subpi.add_argument("--database",
+                       help="The database that will be updated.  Defaults to "
+                            "first database in the regolithrc.json file.")
+    # Do not delete --date arg
+    subpi.add_argument("--date",
+                       help="The date when the institution was created in ISO format. "
+                            "Defaults to today's date."
+                       )
+    return subpi
+
+class InstitutionsUpdaterHelper(DbHelperBase):
+    """
+    Helper for updating/adding  to the projecta collection.
+    """
+    # btype must be the same as helper target in helper.py
+    btype = "u_institutions"
+    needed_dbs = [f'{TARGET_COLL}']
+
+    def construct_global_ctx(self):
+        """Constructs the global context"""
+        super().construct_global_ctx()
+        gtx = self.gtx
+        rc = self.rc
+        rc.coll = f"{TARGET_COLL}"
+        if not rc.database:
+            rc.database = rc.databases[0]["name"]
+        gtx[rc.coll] = sorted(
+            all_docs_from_collection(rc.client, rc.coll), key=_id_key
+        )
+        gtx["all_docs_from_collection"] = all_docs_from_collection
+        gtx["float"] = float
+        gtx["str"] = str
+        gtx["zip"] = zip
+
+    def db_updater(self):
+        rc = self.rc
+        key = rc.institution_id
+        filterid = {'_id': key}
+        target_inst = rc.client.find_one(rc.database, rc.coll, filterid)
+        now = dt.datetime.today()
+        pdoc = {}
+        departments = {}
+        schools = {}
+        if target_inst:
+            if rc.aka:
+                current_aka = target_inst.get('aka')
+                current_aka.extend(rc.aka)
+                pdoc.update({'aka': current_aka})
+            departments = target_inst.get('departments', {})
+            schools = target_inst.get('schools', {})
+        else:
+            inst = fragment_retrieval(self.gtx["institutions"], ["_id"], rc.institution_id)
+            inst.sort(key=lambda x: x['_id'], reverse=False)
+            if not rc.index:
+                print("Please rerun the helper specifying '-n list-index' to update item number 'list-index':")
+                print(f"1. {key} as a new institution.")
+                for i in range(len(inst)):
+                    print(f"{i+2}. {inst[i].get('_id')}      {inst[i].get('name')}.")
+                return
+            if rc.index < 1 or rc.index > len(inst)+1:
+                raise RuntimeError("Sorry, you picked an invalid number.")
+            if rc.index == 1:
+                if not rc.name or not rc.city or not rc.country:
+                    raise RuntimeError("Name, city, and country are required for a new institution.")
+                if rc.country == 'US':
+                    if not rc.zip or not rc.state:
+                        raise RuntimeError("Zip and state are required for a new institution "
+                                           "if institutions is in the US.")
+                pdoc.update({'name': rc.name, 'uuid': str(uuid.uuid4())})
+                if rc.aka:
+                    pdoc.update({'aka':rc.aka})
+                if rc.date:
+                    pdoc.update({'date': rc.date})
+                else:
+                    pdoc.update({'date': now.date()})
+            else:
+                chosen_inst = inst[rc.index-2]
+                key = chosen_inst.get('_id')
+                if rc.aka:
+                    current_aka = chosen_inst.get('aka')
+                    current_aka.extend(rc.aka)
+                    pdoc.update({'aka': current_aka})
+                current_departments = chosen_inst.get('departments')
+                for k, v in current_departments.items():
+                    info_department = {'name':v.get('name'), 'aka':v.get('aka')}
+                    departments.update({k:info_department})
+                current_schools = chosen_inst.get('schools')
+                for k, v in current_schools.items():
+                    info_school = {'name': v.get('name'), 'aka': v.get('aka')}
+                    schools.update({k: info_school})
+        if rc.city:
+            pdoc.update({'city': rc.city})
+        if rc.state:
+            pdoc.update({'state': rc.state})
+        if rc.country:
+            pdoc.update({'country': rc.country})
+        if rc.zip:
+            pdoc.update({'zip': rc.zip})
+        if rc.date:
+            pdoc.update({'updated': rc.date})
+        else:
+            pdoc.update({'updated': now})
+        # departments:
+        if rc.dept_id:
+            dep = {}
+            if rc.dept_id in departments:
+                doc = departments.get(rc.dept_id)
+                current_dept_aka = doc.get('aka')
+                if rc.dept_name:
+                    dep.update({'name': rc.dept_name})
+                if rc.dept_aka:
+                    current_dept_aka.extend(rc.dept_aka)
+                    doc.update({'aka': current_dept_aka})
+                departments.update({rc.dept_id: doc})
+            else:
+                if not rc.dept_name:
+                    raise RuntimeError("Name is required for a new department.")
+                dep.update({'name':rc.dept_name})
+                if rc.dept_aka:
+                    dep.update({'aka': rc.dept_aka})
+                departments[rc.dept_id] = dep
+            pdoc.update({'departments': departments})
+        #schools
+        if rc.school_id:
+            school = {}
+            if rc.school_id in schools:
+                doc = schools.get(rc.school_id)
+                if rc.school_name:
+                    doc.update({'name': rc.school_name})
+                current_sc_aka = doc.get('aka')
+                if rc.dept_aka:
+                    current_sc_aka.extend(rc.school_aka)
+                    doc.update({'aka': current_sc_aka})
+                schools.update({rc.school_id: doc})
+            else:
+                if not rc.school_name:
+                    raise RuntimeError("Name is required for a new school.")
+                school.update({'name': rc.school_name})
+                if rc.school_aka:
+                    school.update({'aka': rc.school_aka})
+                schools[rc.school_id] = school
+                pdoc.update({'schools': schools})
+        rc.client.update_one(rc.database, rc.coll, {'_id': key}, pdoc)
+        print(f"{key} has been updated/added in institutions")
+        return
diff --git a/regolith/schemas.py b/regolith/schemas.py
index 0a537df7..2d14c048 100644
--- a/regolith/schemas.py
+++ b/regolith/schemas.py
@@ -333,7 +333,7 @@ EXEMPLARS = {
                 "aka": ["Chemistry", "Dept. of Chemistry"],
             },
             "apam": {
-                "name": "Department of Applied Physics" "and Applied Mathematics",
+                "name": "Department of Applied Physics " "and Applied Mathematics",
                 "aka": ["APAM"],
             },
         },


Test patch:
diff --git a/tests/outputs/u_institutions/institutions.yaml b/tests/outputs/u_institutions/institutions.yaml
new file mode 100644
index 00000000..0184ff1e
--- /dev/null
+++ b/tests/outputs/u_institutions/institutions.yaml
@@ -0,0 +1,48 @@
+columbiau:
+  aka:
+    - Columbia University
+    - Columbia
+    - ucolumbia
+    - Columbia University in the City of New York
+  city: New York
+  country: USA
+  day: 30
+  departments:
+    physics:
+      name: Department of Physics
+      aka:
+        - Dept. of Physics
+        - Physics
+    chemistry:
+      name: Department of Chemistry
+      aka:
+        - Chemistry
+        - Dept. of Chemistry
+    apam:
+      name: Department of Applied Physics and Applied Mathematics
+      aka:
+        - APAM
+    mathematics:
+      name: Department of Mathematics
+      aka:
+        - dept. of mathematics
+        - math department
+  month: May
+  name: Columbia University
+  schools:
+    seas:
+      name: School of Engineering and Applied Science
+      aka:
+        - SEAS
+        - Columbia Engineering
+        - Fu Foundation School of Engineering and Applied Science
+    cc:
+      name: Columbia College
+      aka:
+        - CC
+  state: NY
+  street: 500 W 120th St
+  updated: '2020-01-01'
+  uuid: avacazdraca345rfsvwre
+  year: 2020
+  zip: '10027'
diff --git a/tests/test_helpers.py b/tests/test_helpers.py
index 84df9ac0..0e6191f5 100644
--- a/tests/test_helpers.py
+++ b/tests/test_helpers.py
@@ -163,8 +163,6 @@ helper_map = [
      "20sb_firstprojectum     status:finished\n"
      "Please rerun the helper specifying the complete ID.\n"
      ),
-
-    #UC1
     (["helper", "lister", "people"],
      "Results of your search:\nsbillinge    \nscopatz\n"),
     (["helper", "lister", "people", "--kv_filter", "name", "simon"],
@@ -193,6 +191,18 @@ helper_map = [
     (["helper", "lister", "people", "--kv_filter", "name", "simon", "position", "singer"],
      "There are no results that match your search.\n"
      ),
+    (["helper", "u_institutions", "columbiau",
+      "--aka", "ucolumbia", "Columbia University in the City of New York",
+      "--dept_id", "mathematics", "--dept_name","Department of Mathematics",
+      "--dept_aka", "dept. of mathematics", "math department",
+      "--school_id", "cc", "--school_name", "Columbia College","--school_aka", "CC",
+      "--date", "2020-01-01"],
+     "columbiau has been updated/added in institutions\n"
+     ),
+    (["helper", "u_institutions", "col"],
+     "Please rerun the helper specifying '-n list-index' to update item number 'list-index':\n"
+     "1. col as a new institution.\n"
+     "2. columbiau      Columbia University.\n"),
 ]