Repository: buildtesters/buildtest (language: python)

Issue:
buildtest tutorial on perlmutter
 In preparation for the buildtest tutorial at ECPAM this PR as a first draft of the **buildtest tutorial on Perlmutter**. The docs was rendered in the PR see https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html or look at the CI checks below

@jscook2345: Can you please help peer-review this MR in preparation for the tutorial.

@wspear: I would like to get your thoughts on this as well. I wasn't sure to what extent we can cover 'E4S Testsuite' and 'Spack Test' on Perlmutter. Note that there will be a hands-on tutorial covering the buildtest-spack integration in https://buildtest.readthedocs.io/en/devel/buildspecs/spack.html that is performed in the container. 


Discussion / hints:
# [Codecov](https://codecov.io/gh/buildtesters/buildtest/pull/1338?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters) Report
Base: **71.03**% // Head: **71.03**% // No change to project coverage :thumbsup:
> Coverage data is based on head [(`98ccb59`)](https://codecov.io/gh/buildtesters/buildtest/pull/1338?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters) compared to base [(`b928070`)](https://codecov.io/gh/buildtesters/buildtest/commit/b928070290495dd3c5512747ae4f29ccde2b1503?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters).
> Patch has no changes to coverable lines.

> :exclamation: Current head 98ccb59 differs from pull request most recent head 9e006cc. Consider uploading reports for the commit 9e006cc to get more accurate results

<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##            devel    #1338   +/-   ##
=======================================
  Coverage   71.03%   71.03%           
=======================================
  Files          57       57           
  Lines        6130     6130           
  Branches     1090     1090           
=======================================
  Hits         4354     4354           
  Misses       1774     1774           
  Partials        2        2           
```



Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters)

</details>

[:umbrella: View full report at Codecov](https://codecov.io/gh/buildtesters/buildtest/pull/1338?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters).   
:loudspeaker: Do you have feedback about the report comment? [Let us know in this issue](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=buildtesters).

Hi @shahzebsiddiqui . I see few changes to be made. 

1. The link to the install buildtest mentioned here is broken I guess. It gives 404 error. See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=Next%2C%20you%20should%20Install%20buildtest%20by%20cloning%20the%20repository%20in%20your%20%24HOME%20directory.

2. The path to clone here https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=HOME%0Agit%20clone-,https%3A//github.com/buildtesters/buildtest,-%24HOME/buildtest%2Dnersc should be for buildtest-nersc repo instead of the buildtest repo.

3. While running the "buildtest build" command in Exercise 1, the path should be "$BUILDTEST_
ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" instead of "perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" ?  See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=perlmutter_tutorial/ex1/hostname.yml%20%2D%2Dpollinterval%3D10
> Hi @shahzebsiddiqui . I see few changes to be made.
> 
> 1. The link to the install buildtest mentioned here is broken I guess. It gives 404 error. See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=Next%2C%20you%20should%20Install%20buildtest%20by%20cloning%20the%20repository%20in%20your%20%24HOME%20directory.
> 2. The path to clone here https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=HOME%0Agit%20clone-,https%3A//github.com/buildtesters/buildtest,-%24HOME/buildtest%2Dnersc should be for buildtest-nersc repo instead of the buildtest repo.
> 3. While running the "buildtest build" command in Exercise 1, the path should be "$BUILDTEST_
>    ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" instead of "perlmutter_tutorial/ex1/hostname.yml --pollinterval=10" ?  See https://buildtest--1338.org.readthedocs.build/en/1338/buildtest_perlmutter.html#:~:text=perlmutter_tutorial/ex1/hostname.yml%20%2D%2Dpollinterval%3D10

thanks for catching these mistakes. I made the corrections. Note you can make comments in-line if you click the review button you can comment directly on the line number. You may find this link https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/reviewing-proposed-changes-in-a-pull-request helpful for reviewing PRs on Github

Fix patch:
diff --git a/docs/builder.rst b/docs/builder.rst
index 0ababf0ad..15a040e76 100644
--- a/docs/builder.rst
+++ b/docs/builder.rst
@@ -49,8 +49,8 @@ is processed.
    :scale: 75 %
 
 
-For every discovered buildspecs, buildtest will validate the buildspecs in the :ref:`parse stage <parse_buildspecs>` to
-ensure buildspec is a valid YAML file according to schema json schema.
+For every discovered buildspec, buildtest will validate the content in the :ref:`parse stage <parse_buildspecs>` to
+ensure buildspec complies to the JSON schema.
 
 .. _parse_buildspecs:
 
diff --git a/docs/features.rst b/docs/features.rst
index ad4768b21..89d85ee1c 100644
--- a/docs/features.rst
+++ b/docs/features.rst
@@ -123,13 +123,11 @@ into CDASH. You must specify a buildname when using **buildtest cdash upload** i
 specify a buildname called `tutorials`::
 
     $ buildtest cdash upload tutorials
-    Reading configuration file:  /Users/siddiq90/Documents/GitHubDesktop/buildtest/buildtest/settings/config.yml
-    Reading report file:  /Users/siddiq90/.buildtest/report.json
-    build name:  tutorials
+    Reading report file:  /Users/siddiq90/Documents/github/buildtest/var/report.json
+    Uploading 110 tests
+    Build Name:  tutorials
     site:  generic
-    stamp:  20210428-1512-Experimental
     MD5SUM: d7651cb3fbdd19298b0188c441704c3a
-    PUT STATUS: 200
     You can view the results at: https://my.cdash.org//viewTest.php?buildid=2004360
 
 We can see the output of these tests in CDASH if we go to url https://my.cdash.org//viewTest.php?buildid=2004360


Test patch:
diff --git a/docs/buildtest_perlmutter.rst b/docs/buildtest_perlmutter.rst
index 6c8462e66..a3c440c4a 100644
--- a/docs/buildtest_perlmutter.rst
+++ b/docs/buildtest_perlmutter.rst
@@ -1,13 +1,13 @@
 Buildtest Tutorial on Perlmutter
 ===================================
 
-This tutorial will be conducted on `Perlmutter <https://docs.nersc.gov/systems/perlmutter/>`_ system. If you need account access please
+This tutorial will be conducted on the `Perlmutter <https://docs.nersc.gov/systems/perlmutter/>`_ system. If you need account access please
 `obtain a user account <https://docs.nersc.gov/accounts/>`_.
 
 Setup
 ------
 
-Once you have a NERSC account, you can `connect to NERSC system <https://docs.nersc.gov/connect/>`_. You will need access to a
+Once you have a NERSC account, you can `connect to any NERSC system <https://docs.nersc.gov/connect/>`_.
 terminal client and ssh into perlmutter as follows::
 
     ssh <user>@perlmutter-p1.nersc.gov
@@ -16,18 +16,22 @@ To get started please load the **python** module since you will need python 3.7
 
     module load python
 
-Next, you should :ref:`Install buildtest <installing_buildtest>` by cloning the repository in your $HOME directory.
-
-Once you have buildtest setup, please clone the following repository https://github.com/buildtesters/buildtest-nersc in your $HOME directory as follows::
+Next, you should :ref:`Install buildtest <installing_buildtest>` by cloning the repository into your home directory::
 
     cd $HOME
+    git clone https://github.com/buildtesters/buildtest.git
+
+Once you have buildtest setup, please clone the following repository into your home directory as follows::
+
     git clone https://github.com/buildtesters/buildtest-nersc $HOME/buildtest-nersc
     export BUILDTEST_CONFIGFILE=$HOME/buildtest-nersc/config.yml
 
-Once you are done, please navigate back to root of buildtest which can be done by running::
+Once you are done, please navigate back to the root of buildtest::
 
     cd $BUILDTEST_ROOT
 
+**If you get stuck on any exercise, you can see the solution to each exercise in file ".solution.txt"**
+
 Exercise 1: Running a Batch Job
 --------------------------------
 
@@ -36,32 +40,55 @@ In this exercise, we will submit a batch job that will run `hostname` in the slu
 .. literalinclude:: ../perlmutter_tutorial/ex1/hostname.yml
    :language: yaml
 
-Let's run this test and poll interval for 10 secs::
+Let's run this test with a poll interval of ten seconds::
 
    buildtest build -b $BUILDTEST_ROOT/perlmutter_tutorial/ex1/hostname.yml --pollinterval=10
 
-Once test is complete, check the output of test by running::
+Once test is complete, check the output of the test by running::
 
     buildtest inspect query -o hostname_perlmutter
 
-Next, let's update the test such that it runs on both **regular** and **debug** queue. You will need to update the **executor** property and
+Next, let's update the test such that it runs on both the **regular** and **debug** queue. You will need to update the **executor** property and
 specify a regular expression. Please refer to :ref:`Multiple Executors <multiple_executors>` for reference. You can retrieve a list of available executors
 by running ``buildtest config executors``.
 
-Once you have updated the test, please rerun the test, now you should expect to see two runs for same test.
+Once you have updated and re-run the test, you should see two test runs for **hostname_perlmutter**, one for each executor. If you ran this successfully, in output of
+``buildtest build`` you should see a test summary with two executors
+
+.. code-block:: console
+
+                                                                    Test Summary
+    ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━┓
+    ┃ builder                               ┃ executor                    ┃ status ┃ checks (ReturnCode, Regex, Runtime) ┃ returncode ┃ runtime  ┃
+    ┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━┩
+    │ hostname_perlmutter/80e317c1          │ perlmutter.slurm.regular    │ PASS   │ N/A N/A N/A                         │ 0          │ 45.324512│
+    ├───────────────────────────────────────┼─────────────────────────────┼────────┼─────────────────────────────────────┼────────────┼──────────┤
+    │ hostname_perlmutter/b1d7b318          │ perlmutter.slurm.debug      │ PASS   │ N/A N/A N/A                         │ 0          │ 75.54278 │
+    └───────────────────────────────────────┴─────────────────────────────┴────────┴─────────────────────────────────────┴────────────┴──────────┘
+
 
 Exercise 2: Performing Status Check
 ------------------------------------
 
-In this exercise, we will check version of Lmod via environment **LMOD_VERSION** and specify the
-the output using :ref:`regular expression <regex>`.
+In this exercise, we will check the version of Lmod using the environment variable **LMOD_VERSION** and specifying the
+the output using a :ref:`regular expression <regex>`. We will run the test with an invalid regular expression and see if test fails and
+rerun example until it passes
 
 .. literalinclude:: ../perlmutter_tutorial/ex2/module_version.yml
    :language: yaml
 
-This buildspec is invalid, your first task is to make sure buildspec is valid. Once you have accomplished this task, try building
-the test and check the output of test. If your test passes, try updating the regular expression and see if test fails. Revert the change
-back and make the test pass.
+First let's try running this test, you will notice the test will fail validation::
+
+    buildtest build -b perlmutter_tutorial/ex2/module_version.yml
+
+
+**TODO:**
+
+- Validate the buildspec using ``buildtest buildspec validate``
+- Add a regular expression on ``stdout`` stream and make sure test fails
+- Check output of test via ``buildtest inspect query``
+- Update regular expression to match output with value of **$LMOD_VERSION** reported in test and rerun test until it passes.
+
 
 Exercise 3: Querying Buildspec Cache
 -------------------------------------
@@ -72,10 +99,12 @@ In this exercise you will learn how to use the :ref:`buildspec_interface`. Let's
 
 In this task you will be required to do the following
 
+**TODO:**
+
 1. Find all tags
-2. List all filter and format fields
-3. Format table via fields ``name``, ``description``
-4. Filter buildspec by tag ``e4s``
+2. List all filters and format fields
+3. Format tables via fields ``name``, ``description``
+4. Filter buildspecs by tag ``e4s``
 5. List all invalid buildspecs
 6. Validate all buildspecs by tag ``e4s``
 7. Show content of test ``hello_world_openmp``
@@ -83,26 +112,40 @@ In this task you will be required to do the following
 Exercise 4: Querying Test Reports
 ----------------------------------
 
-In this exercise you will be learn how to :ref:`query test report <test_reports>`. This can be done by
+In this exercise you will learn how to :ref:`query test reports <test_reports>`. This can be done by
 running ``buildtest report``. In this task please do the following
 
-1. List all filter and format fields
-2. Query all test by returncode 0
-3. Query all test by tag ``e4s``
-4. Print total count of failed tests
+1. List all filters and format fields
+2. Query all tests by returncode 0
+3. Query all tests by tag ``e4s``
+4. Print the total count of all failed tests
 
-Let's upload the test to CDASH by running the following::
+Let's upload the tests to CDASH by running the following::
 
     buildtest cdash upload $USER-buildtest-tutorial
 
-Take some time to analyze the output in CDASH by opening the link including PASS/FAIL test.
+Buildtest :ref:`cdash integration <cdash_integration>` via ``buildtest cdash upload`` allows buildtest to push test results to CDASH server. The test results
+are captured in report file typically shown via ``buildtest report``. CDASH allows one to easily process the test results in web-interface.
+
+If you were successful in running above command, you should see a link to CDASH server https://my.cdash.org with link to test results, please click on the link
+to view your test results and briefly analyze the test results.
+
+.. code-block:: console
+
+       buildtest cdash upload $USER-buildtest-tutorial
+    Reading report file:  /Users/siddiq90/Documents/github/buildtest/var/report.json
+    Uploading 110 tests
+    Build Name:  siddiq90-buildtest-tutorial
+    site:  generic
+    MD5SUM: a589c72bcdabdab9038600a2789e429f
+    You can view the results at: https://my.cdash.org//viewTest.php?buildid=2278337
+
 
 Exercise 5: Specifying Performance Checks
 --------------------------------------------
 
-In this task, we will using :ref:`performance checks <perf_checks>` to determine state of test.
-In this exercise, we will be running the STREAM benchmark. Shown below is an example buildspec that you
-will be working with
+In this task, we will running the STREAM benchmark and use :ref:`performance checks <perf_checks>` to determine if
+test will pass based on the performance results. Shown below is stream example that we will be using for this exercise
 
 .. literalinclude:: ../perlmutter_tutorial/ex5/stream.yml
    :language: yaml
@@ -112,5 +155,8 @@ First, let's build this test and analyze the output::
   buildtest build -b perlmutter_tutorial/ex5/stream.yml
   buildtest inspect query -o stream_test
 
-Take a close look at the metrics value. In this task, you are requested to use use :ref:`assert_ge` with metric ``copy`` and
-``scale`` with reference value. For reference value please experiment with different metrics and see if test pass/fail.
+**TODO**
+
+- Check the output of metrics ``copy`` and ``scale`` in the command **buildtest inspect query -o stream_test**
+- Use the :ref:`assert_ge` check with metric ``copy`` and ``scale``. Specify a reference value (pick some high number) for metric **copy** and **scale*** that will cause test to **FAIL**.
+- Next try different reference values and make sure test will **PASS**.