Repository: google/mobly

Issue:
add a prefix document to test_summary.yaml with requested tests 
This could be dumped before any test cases are run

Discussion:
dthkao: 



Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---

*[mobly/records.py, line 429 at r1](https://reviewable.io:443/reviews/google/mobly/279#-KqFgaA3dUDwcpw4Ygiu:-KqFgaA3dUDwcpw4Ygiv:b-wjrgfc) ([raw file](https://github.com/google/mobly/blob/b6686bb09105a416b5155a0cceac1a06272cc3ec/mobly/records.py#L429)):*
> ```Python
>         return json_str
> 
>     def get_requested_test_names(self):
> ```

What prevents someone from accessing self.requested? (As we do?)

---


*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279)*
<!-- Sent from Reviewable.io -->

xpconanfan: 



Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---

*[mobly/records.py, line 429 at r1](https://reviewable.io:443/reviews/google/mobly/279#-KqFgaA3dUDwcpw4Ygiu:-KqOodMnUi49SWUftmdp:bh7y1z9) ([raw file](https://github.com/google/mobly/blob/b6686bb09105a416b5155a0cceac1a06272cc3ec/mobly/records.py#L429)):*
<details><summary><i>Previously, dthkao (David T.H. Kao) wrote…</i></summary><blockquote>

What prevents someone from accessing self.requested? (As we do?)
</blockquote></details>

test does not have access to the record object.

---


*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279)*
<!-- Sent from Reviewable.io -->

xpconanfan: 



Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---

*[mobly/records.py, line 429 at r1](https://reviewable.io:443/reviews/google/mobly/279#-KqFgaA3dUDwcpw4Ygiu:-KqOptsgc7YNqR5aNy-L:b1fqnmy) ([raw file](https://github.com/google/mobly/blob/b6686bb09105a416b5155a0cceac1a06272cc3ec/mobly/records.py#L429)):*
<details><summary><i>Previously, dthkao (David T.H. Kao) wrote…</i></summary><blockquote>

How/where is this function used, then?
</blockquote></details>

It's used by Mobly itself. Are you seeing all the diffs in your view?

---


*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279)*
<!-- Sent from Reviewable.io -->

dthkao: 



Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---

*[mobly/records.py, line 429 at r1](https://reviewable.io:443/reviews/google/mobly/279#-KqFgaA3dUDwcpw4Ygiu:-KqOr4mFdxohb-QvcJXD:b-dhv434) ([raw file](https://github.com/google/mobly/blob/b6686bb09105a416b5155a0cceac1a06272cc3ec/mobly/records.py#L429)):*
<details><summary><i>Previously, xpconanfan (Ang Li) wrote…</i></summary><blockquote>

It's used by Mobly itself. Are you seeing all the diffs in your view?
</blockquote></details>

I am. There's no new usage of it in the diff?

---


*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279)*
<!-- Sent from Reviewable.io -->

xpconanfan: 



Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---

*[mobly/records.py, line 429 at r1](https://reviewable.io:443/reviews/google/mobly/279#-KqFgaA3dUDwcpw4Ygiu:-KqOrZNzcJvlKNzBYhl-:b-m8finj) ([raw file](https://github.com/google/mobly/blob/b6686bb09105a416b5155a0cceac1a06272cc3ec/mobly/records.py#L429)):*
<details><summary><i>Previously, dthkao (David T.H. Kao) wrote…</i></summary><blockquote>

I am. There's no new usage of it in the diff?
</blockquote></details>

Hmmmm, you're right.
I think the idea behind this was to keep the job of serializing this results object in this class, but I didn't finish the job...

All the other entries are dicts, whereas the test names are just a list.
So we need a key to be associated with the list.
I was gonna put the key in this method, but somehow ended up with the current implementation, sorry...

Will upload a new patch.

---


*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279)*
<!-- Sent from Reviewable.io -->

xpconanfan: 



Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---

*[mobly/records.py, line 429 at r1](https://reviewable.io:443/reviews/google/mobly/279#-KqFgaA3dUDwcpw4Ygiu:-KqOvJCDVXkFJwXFKZeT:b-896fix) ([raw file](https://github.com/google/mobly/blob/b6686bb09105a416b5155a0cceac1a06272cc3ec/mobly/records.py#L429)):*
<details><summary><i>Previously, xpconanfan (Ang Li) wrote…</i></summary><blockquote>

Hmmmm, you're right.
I think the idea behind this was to keep the job of serializing this results object in this class, but I didn't finish the job...

All the other entries are dicts, whereas the test names are just a list.
So we need a key to be associated with the list.
I was gonna put the key in this method, but somehow ended up with the current implementation, sorry...

Will upload a new patch.
</blockquote></details>

Done.

---


*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279)*
<!-- Sent from Reviewable.io -->

dthkao: <img class="emoji" title=":lgtm:" alt=":lgtm:" align="absmiddle" src="https://reviewable.io/lgtm.png" height="20" width="61"/>

---

Review status: 0 of 3 files reviewed at latest revision, 1 unresolved discussion.

---



*Comments from [Reviewable](https://reviewable.io:443/reviews/google/mobly/279#-:-KqOzAiQYu8XZTFQYYdN:bnfp4nl)*
<!-- Sent from Reviewable.io -->


Patch:
diff --git a/mobly/records.py b/mobly/records.py
index 54be008..b6175cd 100644
--- a/mobly/records.py
+++ b/mobly/records.py
@@ -43,8 +43,16 @@ class TestSummaryEntryType(enum.Enum):
     The idea is similar to how `TestResult.json_str` categorizes different
     sections of a `TestResult` object in the serialized format.
     """
+    # A list of all the tests requested for a test run.
+    # This is dumped at the beginning of a summary file so we know what was
+    # requested in case the test is interrupted and the final summary is not.
+    # created.
+    TEST_NAME_LIST = 'TestNameList'
+    # Records of test results.
     RECORD = 'Record'
+    # A summary of the test run stats, like how many test failed.
     SUMMARY = 'Summary'
+    # Information on the controllers used in the test.
     CONTROLLER_INFO = 'ControllerInfo'
 
 
@@ -418,6 +426,17 @@ class TestResult(object):
         json_str = json.dumps(d, indent=4, sort_keys=True)
         return json_str
 
+    def requested_test_names_dict(self):
+        """Gets the requested test names of a test run in a dict format.
+
+        Note a test can be requested multiple times, so there can be duplicated
+        values
+
+        Returns:
+            A dict with a key and the list of strings.
+        """
+        return {'Requested Tests': copy.deepcopy(self.requested)}
+
     def summary_str(self):
         """Gets a string that summarizes the stats of this test result.
 


Test patch:
diff --git a/mobly/base_test.py b/mobly/base_test.py
index ece150a..5deaf5e 100644
--- a/mobly/base_test.py
+++ b/mobly/base_test.py
@@ -552,6 +552,8 @@ class BaseTestClass(object):
                 # No test method specified by user, execute all in test class.
                 test_names = self._get_all_test_names()
         self.results.requested = test_names
+        self.summary_writer.dump(self.results.requested_test_names_dict(),
+                                 records.TestSummaryEntryType.TEST_NAME_LIST)
         tests = self._get_test_methods(test_names)
         # Setup for the class.
         try:
diff --git a/tests/mobly/test_runner_test.py b/tests/mobly/test_runner_test.py
index e57a91f..7690c22 100755
--- a/tests/mobly/test_runner_test.py
+++ b/tests/mobly/test_runner_test.py
@@ -13,11 +13,14 @@
 # limitations under the License.
 
 import mock
+import os
 import shutil
 import tempfile
+import yaml
 from future.tests.base import unittest
 
 from mobly import config_parser
+from mobly import records
 from mobly import signals
 from mobly import test_runner
 
@@ -31,6 +34,7 @@ class TestRunnerTest(unittest.TestCase):
     """This test class has unit tests for the implementation of everything
     under mobly.test_runner.
     """
+
     def setUp(self):
         self.tmp_dir = tempfile.mkdtemp()
         self.base_mock_test_config = config_parser.TestRunConfig()
@@ -50,7 +54,7 @@ class TestRunnerTest(unittest.TestCase):
     def test_register_controller_no_config(self):
         tr = test_runner.TestRunner(self.log_dir, self.test_bed_name)
         with self.assertRaisesRegex(signals.ControllerError,
-                                     'No corresponding config found for'):
+                                    'No corresponding config found for'):
             tr._register_controller(self.base_mock_test_config,
                                     mock_controller)
 
@@ -177,6 +181,37 @@ class TestRunnerTest(unittest.TestCase):
         }
         self.assertEqual(tr.results.controller_info, expected_info)
 
+    def test_summary_file_entries(self):
+        """Verifies the output summary's file format.
+
+        This focuses on the format of the file instead of the content of
+        entries, which is covered in base_test_test.
+        """
+        mock_test_config = self.base_mock_test_config.copy()
+        mock_ctrlr_config_name = mock_controller.MOBLY_CONTROLLER_CONFIG_NAME
+        my_config = [{
+            'serial': 'xxxx',
+            'magic': 'Magic1'
+        }, {
+            'serial': 'xxxx',
+            'magic': 'Magic2'
+        }]
+        mock_test_config.controller_configs[mock_ctrlr_config_name] = my_config
+        tr = test_runner.TestRunner(self.log_dir, self.test_bed_name)
+        tr.add_test_class(mock_test_config, integration_test.IntegrationTest)
+        tr.run()
+        summary_path = os.path.join(mock_test_config.log_path,
+                                    mock_test_config.test_bed_name, 'latest',
+                                    records.OUTPUT_FILE_SUMMARY)
+        with open(summary_path, 'r') as f:
+            summary_entries = list(yaml.load_all(f))
+        self.assertEqual(len(summary_entries), 4)
+        # Verify the first entry is the list of test names.
+        self.assertEqual(summary_entries[0]['Type'],
+                         records.TestSummaryEntryType.TEST_NAME_LIST.value)
+        self.assertEqual(summary_entries[1]['Type'],
+                         records.TestSummaryEntryType.RECORD.value)
+
     @mock.patch(
         'mobly.controllers.android_device_lib.adb.AdbProxy',
         return_value=mock_android_device.MockAdbProxy(1))
@@ -265,8 +300,7 @@ class TestRunnerTest(unittest.TestCase):
 
     def test_run_no_tests(self):
         tr = test_runner.TestRunner(self.log_dir, self.test_bed_name)
-        with self.assertRaisesRegex(test_runner.Error,
-                                     'No tests to execute.'):
+        with self.assertRaisesRegex(test_runner.Error, 'No tests to execute.'):
             tr.run()
 
     def test_verify_controller_module(self):