Repository: innolitics/rdm (language: python)

Issue:
Pulling in documentation improvements
@ReeceStevens I created GitHub Issues for the items we discussed. I also pulled in a few of them already. I trimmed out a few client-specific parts of the V&V plan and also made it line up with the software plan more directly. Let me know what you think.

I also added your GitHub workflow and a couple of other scripts.

If you could review the README, the workflow, the v and v plan, and the updates to the software plan that would be great. There's' no need to review the scripts unless you just want to. In particular, see if the README comments make any sense to you.


Discussion / hints:

Fix patch:
diff --git a/README.md b/README.md
index a46fe2f..04bb4cf 100644
--- a/README.md
+++ b/README.md
@@ -10,6 +10,13 @@ RDM is an open-source documentation as code software tool that provides Markdown
 
 RDM is especially well-suited for early-stage software-only medical devices that are being developed following IEC 62304.
 
+## Who Uses RDM?
+
+- We use it at [Innolitics](https://innolitics).
+- Several of our clients have used it and have successfully submitted 510(k)s using documents produced by it.
+- One client also passed an IEC 62304 [Intertek](https://www.intertek.com) Audit using the documents produced by RDM.
+- Another client uses RDM to manager their entire QMS.
+
 ## Quick Start
 
 ```
@@ -26,9 +33,26 @@ make docs
 
 ## Professional Support
 
-RDM is developed by [Innolitics](https://innolitics.com). We're a small development firm that writes software for medical devices.
+RDM is developed by [Innolitics](https://innolitics.com). We provide engineering and regulatory consulting to medical device startups.
+
+If you need to comply with IEC 62304, we can help. We speak "software" and "regulatory", so we can bridge the gap between regulatory compliance and engineering productivity. Your engineers want to focus on building a high quality product, not learning the details of IEC 62304. We can help you become compliant with minimal distraction.
+
+Our process has three steps:
+
+1. Talk with your tech lead to understand your current, and desired software-development processes
+2. Identify gaps in your IEC-62304 compliance
+3. Work with your engineering team and RA/QA to iteratively fill the gaps
+
+RDM is often a part of how we fill the gaps, although each team's needs are different. We can help you adapt RDM to your unique needs, including developing custom integrations. We can also provide training using lingo and examples your software engineers will understand. Email us at [sales@innolitics.com](mailto:sales@innolitics.com) for pricing and to learn more.
 
-We provide professional support for companies implementing RDM as their regulatory documentation solution. We can provide training, custom integrations, and workflow optimization for your software development team. Email us at [sales@innolitics.com](mailto:sales@innolitics.com) to learn more.
+## Benefits of Using RDM
+
+- Encourages engineers to update documents early in the design process, when they add value, instead of back-filling them just before the next release when they won't add any value.
+- Let your engineers write documents using tools they like (i.e., markdown and YAML)
+- Update and review documentation in parallel to code changes (i.e., in GitHub pull requests)
+- Automatically generated records from data stored in GitHub or other backends
+- Automatically generate traceability matrices (e.g., as part of your CI server)
+- Easily customize and automate your documentation process using custom scripts or data files (see the [Contrib Section](#Contrib) for examples)
 
 ## Our Philosophy on Regulations
 
@@ -135,7 +159,6 @@ The `rdm render` subcommand provides a few extra filters to the Jinja context:
  - join_to: Given a set of ids for an object, and a list of the objects these ids refer to, select out the objects by joining using the specified primary key (which defaults to 'id').
  - md_indent: Processes a snippet of text and removes or adds header indents to match the indent pattern of surrounding text.
 
-
 ### Extensions
 
 We also support [extensions](http://jinja.pocoo.org/docs/2.10/extensions/). Extensions are set using the `md_extensions` configuration paramater in `config.yml`. See the Markdown Extensions section for details about available markdown extensions.
@@ -295,6 +318,46 @@ To provide a custom checklist, use a file path for the first argument.
 
 The checklist format is described in detail [here](./docs/checklist-format.md).
 
+## Contrib
+
+The [contrib folder](https://github.com/innolitics/rdm/tree/main/contrib) includes several scripts and files which may be useful to you. Each is described in some detail here:
+
+### [GitHub Workflow](https://github.com/innolitics/rdm/tree/main/contrib/github_workflow.yml)
+
+One benefit of storing the design history file in the Git repository alongside the code is that you can easily generate the documents for various versions of the code. This GitHub workflow will generate the release PDFs or word documents and store them as artifacts.
+
+### [Simple Requirements Format](https://github.com/innolitics/rdm/tree/main/contrib/convert_requirements.py)
+
+This script lets you write software requirements in a simpler format that looks like this:
+
+```
+1 Hardware
+1.1 First requirements goes here.
+
+2 Loading
+2.1 Second requirement goes here.
+
+3 Users
+3.1 Third requirement goes here.
+3.2 Fourth requirement goes here.
+3.3 User logins
+3.3.1 Fifth requirement goes here.
+3.3.2 Sixth requirement goes here.
+3.4 Seventh requirement goes here.
+```
+
+The script converts this format into a YAML format that can be consumed by the `rdm render` command. This script illustrates that you can customize RDM to your project's unique needs.
+
+### [Download Linked Images](https://github.com/innolitics/rdm/tree/main/contrib/download_images.py)
+
+Markdown lets you include images from web links while latex does not. This script solves this problem. To use it, filter your markdown through it before sending it to pandoc for PDF conversion.
+
+The script parses markdown from stdin, downloads any linked images to directory provided as the first argument, swaps out the URL for the local path, and write the updated markdown to stdout.
+
+Only images linked with a URL with an http or https scheme are included.
+
+The downloaded files retain the extension present in the path portion of the URL, but the name is replaced with the sha256 hash of their contents.
+
 ## RDM's Limitations
 
 - The default templates were written with small software teams in mind (e.g., 2 - 5 developers).
@@ -315,9 +378,4 @@ The checklist format is described in detail [here](./docs/checklist-format.md).
 - Continue to streamline the workflow
 - Provide more thorough examples
 
-## Who Uses RDM?
-
-- We use it at [Innolitics](https://innolitics).
-- A couple of our clients have used it and have successfully submitted 510(k)s using documents produced by it. One client also passed an IEC62304 [Intertek](https://www.intertek.com) Audit using the documents produced by RDM.
-
 **If you use RDM, please let us know.**
diff --git a/contrib/convert_requirements.py b/contrib/convert_requirements.py
new file mode 100644
index 0000000..817bef8
--- /dev/null
+++ b/contrib/convert_requirements.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+import sys
+import collections
+import re
+
+import yaml
+
+
+def read_lines(fileobj):
+    """
+    Split a file into logical lines.
+    """
+    # Lines starting with space are considered continuations
+    return re.sub(r'\n([^ ])', '\x00\\1', fileobj.read()).split('\x00')
+
+
+class DottedDecimal(collections.namedtuple("DottedDecimal", ("path"))):
+    """
+    Arbitrary-length path of decimals, like a version number.
+    1.1, 1..1 and 1.1. are considered equivalent
+    """
+
+    @classmethod
+    def from_text(cls, text):
+        return cls(tuple(int(p) for p in text.split(".") if p))
+
+    @property
+    def parent(self):
+        parent_path = self.path[:-1]
+        return DottedDecimal(parent_path) if parent_path else None
+
+    def __str__(self):
+        return ".".join(str(i) for i in self.path)
+
+
+def parse_requirements(fileobj):
+    """
+    Parse lines from a file into a dictionary
+    """
+    lines = read_lines(fileobj)
+    elements = collections.OrderedDict()
+    parents = set()  # Set of keys known to appear above another key
+    for line in lines:
+        key_str, text = line.split(None, 1)
+        key = DottedDecimal.from_text(key_str)
+        if key.parent:
+            parents.add(key.parent)
+        if key not in elements:
+            elements[key] = collections.OrderedDict([("text", text.strip())])
+        else:
+            print(f"The id '{key}' occurs more than once", file=sys.stderr)
+            sys.exit(1)
+    for key, properties in elements.items():
+        if key in parents:
+            properties["type"] = "section"
+    return elements
+
+
+def parse_requirements_flat(fileobj):
+    records = []
+    for key, properties in parse_requirements(fileobj).items():
+        req_id = f'REQ-{key}'
+        record = {**properties, "id": req_id}
+        if key.parent:
+            record["parent"] = str(key.parent)
+        records.append(record)
+    return records
+
+
+if __name__ == "__main__":
+    requirements = parse_requirements_flat(fileobj=sys.stdin)
+    yaml.dump(requirements, sys.stdout)
diff --git a/contrib/download_images.py b/contrib/download_images.py
new file mode 100644
index 0000000..52b8848
--- /dev/null
+++ b/contrib/download_images.py
@@ -0,0 +1,58 @@
+#!/usr/bin/env python3
+import re
+import sys
+import urllib.request
+import hashlib
+import os.path
+from urllib.parse import urlparse
+
+import argparse
+
+
+def download_images_and_sub_urls(line, download_to):
+    # TODO: confirm that this works if there are two images on one line
+    markdown_linked_image = re.compile(r"!\[.*]\((?P<image_url>.*)\)")
+    for match in markdown_linked_image.finditer(line):
+        try:
+            image_url = urlparse(match.group('image_url'))
+            image_url_path = image_url.path
+            image_url_scheme = image_url.scheme
+        except ValueError:
+            print(f"Skipping invalid URL: {match.group('image_url')}", file=sys.stderr)
+            continue
+        if image_url_scheme not in ["http", "https"]:
+            continue
+        _, file_extension = os.path.splitext(image_url_path)
+        if file_extension == '':
+            print(f"Skipping URL with no file extension: {image_url.geturl()}", file=sys.stderr)
+            continue
+
+        with urllib.request.urlopn(image_url.geturl()) as f:
+            image_data = f.read()
+        hasher = hashlib.sha256()
+        hasher.update(image_data)
+        image_hash = hasher.hexdigest()
+        downloaded_image_path = os.path.join(download_to, f"{image_hash}{file_extension}")
+        with open(downloaded_image_path, 'wb') as f:
+            f.write(image_data)
+        print(f'Downloaded {image_url.geturl()} to {downloaded_image_path}', file=sys.stderr)
+        line = line[:match.start('image_url')] + downloaded_image_path + line[match.end('image_url'):]
+    return line
+
+
+if __name__ == "__main__":
+    description = '''
+    Parse markdown from stdin, download any linked images to a designated
+    directory, swap out the URL for the local path, and write the updated
+    markdown to stdout.  Only images linked with a URL with an http or https
+    scheme are included.  The downloaded files retain the extension present in
+    the path portion of the URL, but the name is replaced with the sha256 hash
+    of their contents.
+    '''
+    parser = argparse.ArgumentParser(description=description)
+    parser.add_argument('download_to', help='path to directory where images will be downloaded')
+    args = parser.parse_args()
+
+    for line in sys.stdin.readlines():
+        updated_line = download_images_and_sub_urls(line, args.download_to)
+        print(updated_line, end='')
diff --git a/contrib/github_workflow.yml b/contrib/github_workflow.yml
new file mode 100644
index 0000000..15e24d1
--- /dev/null
+++ b/contrib/github_workflow.yml
@@ -0,0 +1,40 @@
+name: Regulatory Document Generation
+on: [workflow_dispatch]
+jobs:
+  rdm-generate-docs:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Check out repository code
+        uses: actions/checkout@v2
+      - name: Set up Python 3.8
+        uses: actions/setup-python@v1
+        with:
+          python-version: 3.8
+
+      - name: Install RDM dependencies
+        run: |
+          sudo apt install pandoc texlive-latex-extra latexmk
+          pip3 install --user virtualenv
+          python3 -m virtualenv tmp-venv
+          source tmp-venv/bin/activate
+          pip install --no-cache-dir -r requirements.txt
+      - name: Generate release documents
+        run: |
+          source tmp-venv/bin/activate
+          pushd regulatory
+          make all
+          make pdfs
+          make docs
+          popd
+      # For now, PDFs are not in use
+      # - name: Store PDFs as artifacts
+      #   uses: actions/upload-artifact@v2
+      #   with:
+      #       name: regulatory-pdfs.zip
+      #       path: ./release/*.pdf
+
+      - name: Store docx files as artifacts
+        uses: actions/upload-artifact@v2
+        with:
+            name: regulatory-docs.zip
+            path: regulatory/release/*.docx
diff --git a/rdm/init_files/documents/software_plan.md b/rdm/init_files/documents/software_plan.md
index 61649b5..50f5690 100644
--- a/rdm/init_files/documents/software_plan.md
+++ b/rdm/init_files/documents/software_plan.md
@@ -87,13 +87,7 @@ To the extent possible, checking against these standards should be performed in
 
 ## Testing Plan
 
-[[62304:5.5.2]]
-
-All final tests must include the Git hash or other objective reference that can be used to identify the exact software tested [[62304:5.1.11]].
-
-TODO: Write out a testing plan for {{ device.name }}.
-
-This plan should include a pass/fail criteria for the entire test suite.  E.g., you require that all unit tests pass and that all integration tests pass or the cause of the failure is understood and justified [[62304:5.7.1.a]]
+See the Verification and Validation Plan (VVP-001).
 
 ## Risk Management
 
diff --git a/rdm/init_files/documents/verification_and_validation_plan.md b/rdm/init_files/documents/verification_and_validation_plan.md
new file mode 100644
index 0000000..6fca975
--- /dev/null
+++ b/rdm/init_files/documents/verification_and_validation_plan.md
@@ -0,0 +1,75 @@
+---
+id: VVP-001
+revision: 1
+title: Verification and Validation Plan
+---
+
+# Purpose
+
+This document describes the plan regarding how {{ device.name }} will be verified and validated.
+
+[[Fulfills FDA-CPSSCMD:v-and-v and 62304:5.5.2]]
+
+# Definitions
+
+**Verification** means confirmation by examination and provision of objective evidence that specified requirements have been fulfilled. In other words, "Did we build the thing right?"
+
+**Validation** means establishing by objective evidence that device specifications conform with user needs and intended uses. In other words, "Did we build the right thing?"
+
+An **end-to-end test** examines the functionality of an software system through its standard user interface without peering into its internal structures or workings.
+
+A **unit test** examines an individual software unit that typically cannot be broken down any further.
+
+An **integration test** examines multiple software units for defects in the interfaces and interactions between them.
+
+An **automated test** is any test that can be run without human input. A **manual test** is any test that requires a human to perform some or all of the test steps.
+
+A **software item** is any identifiable part of a computer program.
+
+A **software system** is an integrated collection of software items organized to accomplish a specific function or set of functions.
+
+A **software unit** is a software item that is not subdivided into other software items.
+
+**Regression testing** is the process of re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change.
+
+# Verification Activities
+
+Software verification begins with the code reviews and adherence to software standards defined in the Software Plan. The outputs of these activities are recorded in the Revision Level History.
+
+Software verification continues by running all automated tests on a continuous basis during the development process to detect any regressions. Automated tests are automatically run against each proposed software revision before the revision is accepted, a process known as Continuous Integration, or CI. Manual tests are run as needed during development and then for the Software Test Record for a release.
+
+Tests are written, modified, and reviewed during the Unit Implementation and Testing activity and are reviewed for completeness as part of the Release activity. Both activities are described in the Software Plan.
+
+A detailed list of verification tests, the testing environment, and their results are found in the Software Test Record. The Software Test Record is created as part of the Integration and System Testing activity described in the Software Plan.
+
+The types of tests used to verify {{ device.name }} are described below:
+
+## Unit Tests
+
+TODO: Describe how and when unit test shall be written
+
+## Integration Tests
+
+TODO: Describe how and when integration tests shall be written
+
+## Automated End-to-End Tests
+
+TODO: Describe how and when automated end-to-end tests shall be written
+
+## Manual End-to-End Tests
+
+TODO: Describe how and when manual end-to-end tests shall be written
+
+# Validation Activities
+
+TODO: Describe how we'll validate that the user needs have been met (in software and hardware projects, this is usually handled at the system-level).
+
+# Pass / Fail Criteria
+
+TODO:
+
+This plan should include a pass/fail criteria for the entire test suite.  E.g., you require that all unit tests pass and that all integration tests pass or the cause of the failure is understood and justified
+
+ENDTODO
+
+[[62304:5.7.1.a]]
diff --git a/rdm/version.py b/rdm/version.py
index b950a80..f323a57 100644
--- a/rdm/version.py
+++ b/rdm/version.py
@@ -1,1 +1,1 @@
-__version__='0.11.0'
\ No newline at end of file
+__version__ = '0.11.0'


Test patch:
diff --git a/rdm/init_files/documents/test_record.md b/rdm/init_files/documents/test_record.md
index c73fa2d..14e5bd5 100644
--- a/rdm/init_files/documents/test_record.md
+++ b/rdm/init_files/documents/test_record.md
@@ -11,6 +11,7 @@ The purpose of this document is to record the status of the software tests run f
 
 The scope of this document is the software system within the {{ device.name }} product.
 
+
 # Verification
 
 I, FULL DEVELOPER NAME, verify that the results recorded here are complete and accurate [[62304:9.8.g]].
@@ -24,11 +25,12 @@ The tests meet our specified pass fail criteria (see Test Plan section of the So
 
 # Test Environment
 
-TODO: Describe the test environment.  This section should include all of the information necessary for someone to reproduce the tests.  For example, it could be wise to include a list of all the environment variables, installed system packages and versions, the git hash, hardware used, etc.  It should also include any relevant testing tools.
+TODO: Describe the test environment.  This section should include all of the information necessary for someone to reproduce the tests.  For example, it could be wise to include a list of all the environment variables, installed system packages and versions, the git hash, hardware used, etc.  It should also include any relevant testing tools [[62304:5.1.11]].
 TODO: document the software version tested [[62304:9.8.c]]
 TODO: document any relevant configuration [[62304:9.8.d]]
 TODO: document the relevant tools used to run these tests [[62304:9.8.e]]
 
+
 # Test Results
 
 [[These are the results of automated unit and integration testing as well as manual testing 62304:9.8.a]]