Repository: regro/regolith

Issue:
helper: milestone updater
add or change a milestone, including deliverable/kickoff milestones.

Discussion:
dy2403: it is passing tests locally, I am trying to understand why there are assert errors here. 
codecov[bot]: # [Codecov](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=h1) Report
> Merging [#439](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=desc) into [master](https://codecov.io/gh/regro/regolith/commit/ed0a79dd6ad1cb98a77632b65ebe61135c10dffd&el=desc) will **increase** coverage by `0.51%`.
> The diff coverage is `83.33%`.

[![Impacted file tree graph](https://codecov.io/gh/regro/regolith/pull/439/graphs/tree.svg?width=650&height=150&src=pr&token=YOWfAqEPVS)](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #439      +/-   ##
==========================================
+ Coverage   62.82%   63.33%   +0.51%     
==========================================
  Files          50       51       +1     
  Lines        4551     4659     +108     
==========================================
+ Hits         2859     2951      +92     
- Misses       1692     1708      +16     
```


| [Impacted Files](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [regolith/helpers/u\_milestonehelper.py](https://codecov.io/gh/regro/regolith/pull/439/diff?src=pr&el=tree#diff-cmVnb2xpdGgvaGVscGVycy91X21pbGVzdG9uZWhlbHBlci5weQ==) | `83.17% <83.17%> (ø)` | |
| [regolith/helper.py](https://codecov.io/gh/regro/regolith/pull/439/diff?src=pr&el=tree#diff-cmVnb2xpdGgvaGVscGVyLnB5) | `100.00% <100.00%> (ø)` | |
| [regolith/fsclient.py](https://codecov.io/gh/regro/regolith/pull/439/diff?src=pr&el=tree#diff-cmVnb2xpdGgvZnNjbGllbnQucHk=) | `77.84% <0.00%> (+1.19%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=footer). Last update [ed0a79d...9a2baff](https://codecov.io/gh/regro/regolith/pull/439?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

sbillinge: ok sounds good. If there are two tests you are good, I just didn't see it.

> Should I be worried about the failing test on CI? It is passing locally,
and I think it is the lister that is failing here.
yes you should!

You can probably make it fail locally by rebasing.  It should be an easy
rebase because there are no conflicts. check out master, update master
checkout your branch run rebase.

On Thu, Jun 18, 2020 at 8:30 AM dy2403 <notifications@github.com> wrote:

> *@dy2403* commented on this pull request.
> ------------------------------
>
> In tests/test_helpers.py
> <https://github.com/regro/regolith/pull/439#discussion_r442189344>:
>
> >       ),
> -    # # this works! adding a new milestone:
>
> Thanks, professor!
> The test in line 59 is updating the milestone number 4 (due date and
> status), so there are two tests, and the test on line 63 is listing. Do you
> mean adding one more?
> I will test it in rg-db-group! I would be happy to roll it out in the
> group meeting!
> Should I be worried about the failing test on CI? It is passing locally,
> and I think it is the lister that is failing here.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/regro/regolith/pull/439#discussion_r442189344>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABAOWUOQZNVKEFWXY53K55LRXICH5ANCNFSM4N6S3XCA>
> .
>


-- 
Professor Simon Billinge
Columbia University


Patch:
diff --git a/regolith/helper.py b/regolith/helper.py
index bf1210c3..843522a7 100644
--- a/regolith/helper.py
+++ b/regolith/helper.py
@@ -10,6 +10,7 @@ from regolith.helpers import l_projectahelper as l_projecta
 from regolith.helpers import l_grantshelper as l_grants
 from regolith.helpers import l_membershelper as l_members
 from regolith.helpers import l_contactshelper as l_contacts
+from regolith.helpers import u_milestonehelper as u_milestone
 
 from regolith.helpers import u_contacthelper as u_contact
 HELPERS = {
@@ -24,6 +25,7 @@ HELPERS = {
     "l_members": (l_members.MembersListerHelper, l_members.subparser),
     "l_contacts": (l_contacts.ContactsListerHelper, l_contacts.subparser),
     "u_contact": (u_contact.ContactUpdaterHelper, u_contact.subparser),
+    "u_milestone": (u_milestone.MilestoneUpdaterHelper, u_milestone.subparser),
 }
 
 
diff --git a/regolith/helpers/u_milestonehelper.py b/regolith/helpers/u_milestonehelper.py
new file mode 100644
index 00000000..affe269b
--- /dev/null
+++ b/regolith/helpers/u_milestonehelper.py
@@ -0,0 +1,146 @@
+"""Helper for updating milestones to the projecta collection.
+    It can update the status and due date of a projectum.
+    It can add a new milestone to the projecta collection.
+"""
+import datetime as dt
+import sys
+from dateutil import parser
+from regolith.helpers.basehelper import DbHelperBase
+from regolith.fsclient import _id_key
+from regolith.tools import all_docs_from_collection
+
+
+TARGET_COLL = "projecta"
+ALLOWED_TYPES = {"m":"meeting", "r":"release", "p":"pull request", "o":"other"}
+ALLOWED_STATUS = {"p":"proposed", "s":"started", "f":"finished", "b":"back_burner","c":"converged"}
+
+def subparser(subpi):
+    subpi.add_argument("projectum_id", help="the id of the projectum")
+    subpi.add_argument("--number",
+                        help="number of the milestone to update from numbered list")
+    subpi.add_argument("--due_date",
+                       help="new due date of the milestone in ISO format (YYYY-MM-DD) "
+                            "required to add a new milestone")
+    subpi.add_argument("--name",
+                       help="name of the new milestone. "
+                            "required to add a new milestone")
+    subpi.add_argument("-o", "--objective",
+                       help="objective of the new milestone. "
+                            "required to add a new milestone")
+    subpi.add_argument("-s", "--status",
+                       help="initial of the status of the milestone/deliverable: "
+                            f"{ALLOWED_STATUS}")
+    subpi.add_argument("-t", "--type",
+                       help="initial of type of the new milestone "
+                            f"{ALLOWED_TYPES}")
+    # Do not delete --database arg
+    subpi.add_argument("--database",
+                       help="The database that will be updated.  Defaults to "
+                            "first database in the regolithrc.json file.")
+    return subpi
+
+class MilestoneUpdaterHelper(DbHelperBase):
+    """Helper for updating milestones to the projecta collection.
+    """
+    # btype must be the same as helper target in helper.py
+    btype = "u_milestone"
+    needed_dbs = [f'{TARGET_COLL}']
+
+    def construct_global_ctx(self):
+        """Constructs the global context"""
+        super().construct_global_ctx()
+        gtx = self.gtx
+        rc = self.rc
+        rc.coll = f"{TARGET_COLL}"
+        if not rc.database:
+            rc.database = rc.databases[0]["name"]
+        gtx[rc.coll] = sorted(
+            all_docs_from_collection(rc.client, rc.coll), key=_id_key
+        )
+        gtx["all_docs_from_collection"] = all_docs_from_collection
+        gtx["float"] = float
+        gtx["str"] = str
+        gtx["zip"] = zip
+
+    def db_updater(self):
+        rc = self.rc
+        key = rc.projectum_id
+        coll = self.gtx[rc.coll]
+        pdocl = list(filter(lambda doc: doc["_id"] == key, coll))
+        if len(pdocl) == 0:
+            raise RuntimeError(
+                "This entry appears to not exist in the collection")
+        filterid = {'_id': key}
+        current = rc.client.find_one(rc.database, rc.coll, filterid)
+        milestones = current.get('milestones')
+        deliverable = current.get('deliverable')
+        kickoff = current.get('kickoff')
+        deliverable['identifier'] = 'deliverable'
+        kickoff['identifier'] = 'kickoff'
+        for item in milestones:
+            item['identifier'] = 'milestones'
+        all_milestones = [deliverable, kickoff]
+        all_milestones.extend(milestones)
+        all_milestones.sort(key=lambda x: x['due_date'], reverse=False)
+        index = 2
+        numbered_milestones = {}
+        for item in all_milestones:
+            numbered_milestones[str(index)] = item
+            index += 1
+        if not rc.number:
+            print("Please choose from one of the following to update/add:")
+            print("1. new milestone")
+            for i in numbered_milestones:
+                current_mil = numbered_milestones[i]
+                if 'name' in current_mil:
+                    print("{}. {}    due date: {}    {}".format(i, current_mil["name"],
+                                                    current_mil["due_date"], current_mil["status"]))
+                else:
+                    print("{}. {}    due date: {}    {}".format(i, current_mil['identifier'],
+                                                                current_mil["due_date"], current_mil["status"]))
+                del current_mil['identifier']
+            return
+        pdoc = {}
+        if int(rc.number) == 1:
+            mil = {}
+            if not rc.due_date or not rc.name or not rc.objective:
+                for i in numbered_milestones:
+                    current_mil = numbered_milestones[i]
+                    del current_mil['identifier']
+                raise RuntimeError("name, objective, and due date are required for a new milestone")
+            mil.update({'due_date': rc.due_date, 'objective': rc.objective, 'name': rc.name})
+            mil.update({'audience': ['lead', 'pi', 'group_members']})
+            if rc.status:
+                mil.update({'status': ALLOWED_STATUS[rc.status]})
+            else:
+                mil.update({'status': 'proposed'})
+            if rc.type:
+                mil.update({'type': ALLOWED_TYPES[rc.type]})
+            else:
+                mil.update({'type': 'meeting'})
+            milestones.append(mil)
+            pdoc = {'milestones':milestones}
+        if int(rc.number) > 1:
+            doc = numbered_milestones[rc.number]
+            identifier = doc['identifier']
+            if rc.due_date:
+                doc.update({'due_date':rc.due_date})
+            if rc.status:
+                doc.update({'status': ALLOWED_STATUS[rc.status]})
+            if identifier == 'milestones':
+                new_mil = []
+                for i in numbered_milestones:
+                    if numbered_milestones[i]['identifier'] =='milestones' and i!= rc.number:
+                        new_mil.append(numbered_milestones[i])
+                new_mil.append(doc)
+                pdoc.update({'milestones':new_mil})
+            else:
+                pdoc.update({identifier:doc})
+        for i in numbered_milestones:
+            current_mil = numbered_milestones[i]
+            del current_mil['identifier']
+
+        rc.client.update_one(rc.database, rc.coll, filterid, pdoc)
+        print("{} has been updated in projecta".format(key))
+
+        return
diff --git a/regolith/schemas.py b/regolith/schemas.py
index 9e481f4b..3fd638a8 100644
--- a/regolith/schemas.py
+++ b/regolith/schemas.py
@@ -1072,15 +1072,7 @@ EXEMPLARS = {
         "lead": "ascopatz",
         "log_url": "https://docs.google.com/document/d/1YC_wtW5Q",
         "milestones": [{
-            'date': '2020-05-05',
-            'due_date': '2020-05-06',
-            'name': 'Kick off meeting',
-            'objective': 'roll out of project to team',
-            'audience': ['lead', 'pi', 'group_members',
-                         'collaborators'],
-            'status': 'finished'
-        },
-            {'due_date': '2020-05-20',
+            'due_date': '2020-05-20',
              'name': 'Project lead presentation',
              'objective': 'lead presents background reading and '
                           'initial project plan',
@@ -1291,7 +1283,7 @@ EXEMPLARS = {
             "status": "accepted",
             "team": [
                 {
-                    "institution": "Columbia Unviersity",
+                    "institution": "Columbia University",
                     "name": "qdu",
                     "position": "co-pi",
                 },


Test patch:
diff --git a/tests/outputs/a_projectum/projecta.yaml b/tests/outputs/a_projectum/projecta.yaml
index c75d37dd..31c4a3ef 100644
--- a/tests/outputs/a_projectum/projecta.yaml
+++ b/tests/outputs/a_projectum/projecta.yaml
@@ -84,16 +84,6 @@
   lead: ascopatz
   log_url: https://docs.google.com/document/d/1YC_wtW5Q
   milestones:
-    - date: '2020-05-05'
-      due_date: '2020-05-06'
-      name: Kick off meeting
-      objective: roll out of project to team
-      audience:
-        - lead
-        - pi
-        - group_members
-        - collaborators
-      status: finished
     - due_date: '2020-05-20'
       name: Project lead presentation
       objective: lead presents background reading and initial project plan
diff --git a/tests/outputs/a_proposal/proposals.yaml b/tests/outputs/a_proposal/proposals.yaml
index d8e88099..2ba5997d 100644
--- a/tests/outputs/a_proposal/proposals.yaml
+++ b/tests/outputs/a_proposal/proposals.yaml
@@ -53,15 +53,15 @@ dmref15:
   pi: Simon Billinge
   status: accepted
   team:
-    - institution: Columbia Unviersity
+    - institution: Columbia University
       name: qdu
-      position: Co-PI
-    - institution: Columbia Unviersity
+      position: co-pi
+    - institution: Columbia University
       name: dhsu
-      position: Co-PI
-    - institution: Columbia Unviersity
+      position: co-pi
+    - institution: Columbia University
       name: sbillinge
-      position: PI
+      position: pi
       subaward_amount: 330000.0
   title: 'DMREF: Novel, data validated, nanostructure determination methods for accelerating
     materials discovery'
diff --git a/tests/outputs/u_milestone/projecta.yaml b/tests/outputs/u_milestone/projecta.yaml
new file mode 100644
index 00000000..d2ef308a
--- /dev/null
+++ b/tests/outputs/u_milestone/projecta.yaml
@@ -0,0 +1,107 @@
+20ly_newprojectum:
+  begin_date: 2020-04-29
+  collaborators:
+    - afriend
+  deliverable:
+    due_date: 2021-01-01
+    audience:
+      - beginning grad in chemistry
+    success_def: audience is happy
+    scope:
+      - UCs that are supported or some other scope description if it software
+      - sketch of science story if it is paper
+    platform: description of how and where the audience will access the deliverable.  journal
+      if it is a paper
+    roll_out:
+      - steps that the audience will take to access and interact with the deliverable
+      - not needed for paper submissions
+    status: proposed
+  description: more work
+  grants:
+    - SymPy-1.1
+  group_members:
+    - ascopatz
+  kickoff:
+    due_date: 2020-05-06
+    audience:
+      - lead
+      - pi
+      - group_members
+    name: Kick off meeting
+    objective: introduce project to the lead
+    status: proposed
+  lead: lyang
+  log_url: ''
+  milestones:
+    - due_date: 2020-05-20
+      name: Project lead presentation
+      objective: to act as an example milestone.  The date is the date it was finished.  delete
+        the field until it is finished.  In this case, the lead will present what
+        they think is the project after their reading. Add more milestones as needed.
+      audience:
+        - lead
+        - pi
+        - group_members
+      status: proposed
+      type: meeting
+  name: New projectum
+  pi_id: scopatz
+  status: started
+20sb_firstprojectum:
+  begin_date: '2020-04-28'
+  collaborators:
+    - aeinstein
+    - pdirac
+  deliverable:
+    audience:
+      - beginning grad in chemistry
+    due_date: '2021-05-05'
+    success_def: audience is happy
+    scope:
+      - UCs that are supported or some other scope description if it is software
+      - sketch of science story if it is paper
+    platform: description of how and where the audience will access the deliverable.  Journal
+      if it is a paper
+    roll_out:
+      - steps that the audience will take to access and interact with the deliverable
+      - not needed for paper submissions
+    status: finalized
+  description: My first projectum
+  end_date: '2020-06-05'
+  grants: SymPy-1.1
+  group_members:
+    - ascopatz
+  kickoff:
+    date: '2020-05-05'
+    due_date: '2020-05-06'
+    name: Kick off meeting
+    objective: introduce project to the lead
+    audience:
+      - lead
+      - pi
+      - group_members
+    status: finished
+  lead: ascopatz
+  log_url: https://docs.google.com/document/d/1YC_wtW5Q
+  milestones:
+    - due_date: '2020-05-20'
+      name: Project lead presentation
+      objective: lead presents background reading and initial project plan
+      audience:
+        - lead
+        - pi
+        - group_members
+      status: proposed
+      type: meeting
+    - due_date: '2020-05-30'
+      name: planning meeting
+      objective: develop a detailed plan with dates
+      audience:
+        - lead
+        - pi
+        - group_members
+      status: converged
+      type: pr
+  name: First Projectum
+  pi_id: scopatz
+  status: proposed
diff --git a/tests/test_helpers.py b/tests/test_helpers.py
index e2d57ae0..549920ae 100644
--- a/tests/test_helpers.py
+++ b/tests/test_helpers.py
@@ -61,7 +61,20 @@ helper_map = [
     (["helper", "l_contacts", "run", "-n", "ny", "-i", "col", "-o", "coffee", "-d", "2019-01-15", "-r", "2"],
      ""
      ),
-    ]
+    (["helper", "u_milestone", "20sb_firstprojectum", "--number", "4",
+      "-s", "c", "--due_date", "2020-05-30"],
+     "20sb_firstprojectum has been updated in projecta\n"
+     ),
+    (["helper", "u_milestone", "20sb_firstprojectum"],
+     "Please choose from one of the following to update/add:\n"
+     "1. new milestone\n"
+     "2. Kick off meeting    due date: 2020-05-06    finished\n"
+     "3. Project lead presentation    due date: 2020-05-20    proposed\n"
+     "4. planning meeting    due date: 2020-05-30    converged\n"
+     "5. deliverable    due date: 2021-05-05    finalized\n"
+     ),
+]
+
 
 
 @pytest.mark.parametrize("hm", helper_map)
diff --git a/tests/u_milestone.rst b/tests/u_milestone.rst
new file mode 100644
index 00000000..b40f6681
--- /dev/null
+++ b/tests/u_milestone.rst
@@ -0,0 +1,12 @@
+**Added:**
+ * a milestone helper for adding/updating milestones/deliverable to the milestones collection.
+
+**Changed:** None
+
+**Deprecated:** None
+
+**Removed:** None
+
+**Fixed:** None
+
+**Security:** None