Repository: MycroftAI/mycroft-core

Issue:
Voight Kampff should allow setting unit
For https://github.com/TheLastProject/travel-time-skill, I need to be able to test both unit settings.

I could not find any code that allows setting the unit.

Being able to use `Given the unit system is imperial` and `Given the unit system is metric` would be great.

(I also noticed https://mycroft-ai.gitbook.io/docs/skill-development/introduction-1/test-steps#and-but names the text `Given user is located in Paris`, but I can't find any code implementing this either?)


Discussion:
Example of what I want to do:

```
Feature: travel-time
  Scenario: Traffic to Berlin
    Given an English speaking user
    Given the unit system is Metric
    Given home is set to Amsterdam
    When the user says "What's the traffic like to Berlin"
    Then mycroft should reply with a dialog from "time.travel.metric"
    Then the reply should contain "from Amsterdam to Berlin by car"
```

I expect this to be equivalent to
```
{
  "utterance": "What's the traffic like to Berlin",
  "intent_type": "time.travel.intent",
  "expected_dialog": "time.travel.metric",
  "expected_response": "It is .* kilometers and .* minutes from Amsterdam to berlin by car",
  "settings": {"home": "Amsterdam"}
}
```
with Mycroft's unit system set to metric as well.
Hi, setting the unit (or any global config) is a good suggestion and is possible in the current system I'll make a point to implement a given step for this.

The settings are a bit trickier but that sort of feature is planned for the second stage of Voight Kampff development. Stage 1 was development of a robust integration test to allow pull requests to Mycroft core and it's supporting modules to be thoroughly tested before inclusion. Stage 2 is meant to implement the features in the current skill tester to allow it to mock parts in the same way as the old skill t ester and apply settings, etc.
I agree with adding to the config abilities and believe there should be a consistent template used as we grow this functionality. Opened issue #2556 to cover this.
I'm trying to finalize the PR for this and wonder what the best syntax in the behave file will be.

For the unit specific given I'm using the suggested: `Given the unit system is X`

For generic configuration I wonder if we can use a dot-notation
`Given the config[uration] value for skills.msm.directory is skills`?


As I recommended in #2556, I would suggest using:

```python3
@given("the skill's {key} is {value}")
def given_skill_config(context, key, value):
```

This aligns with the work I am doing to standardize other language in VK tests. Eg:

``` python3
@given("the user's {key} is {value}")
def given_user_config(context, key, value):
    if key == 'language':
        if value in ['en-us', 'english']:
            context.lang = 'en-us'
            return
        raise ValueError('"{}" is not an allowed language'.format(value))
    raise ValueError('"{}" is not an allowed user configuration ' +
                     'variable'.format(key))
```
In this instance the measurement system is normally set per user so I'd go with:
`"Given the user's {key} is {value}"`
> Given the user's measurement unit is metric

Could use dot notation, or a simple dictionary to map the natural language `{key}` to the relevant dot notation. This would read better (eg above), but does introduce another step for error. Eg was it "measurement unit", "unit system", "unit of measurement", etc 

A dict like this could be useful for other features, like the Config Skill letting users change or query any of their settings. 
Sounds like a good idea, Where would this dict be defined?
`mycroft/res/text/en-us/settings.list`
?
I agree a key / val file sounds like a good idea.

It probably makes sense to have two different files; one in mycroft-core, which defines defaults, and one located in the skill-specific folder to override / provide skill-specific keywords. This would allow us to write mycroft-core-wide statements (eg: @given("the skill's {key} is {value}")) that could then be easily extensible in all skill tests.

As it stands / without this, a skill creator would be forced to copy the function from mycroft-core into their own specific skill's tests to add a custom given statement.
Sounds like a good idea, I'll update my original work to include these features

Resolving patch:
diff --git a/mycroft/configuration/config.py b/mycroft/configuration/config.py
--- a/mycroft/configuration/config.py
+++ b/mycroft/configuration/config.py
@@ -28,8 +28,8 @@
 
 
 def is_remote_list(values):
-    ''' check if this list corresponds to a backend formatted collection of
-    dictionaries '''
+    """Check if list corresponds to a backend formatted collection of dicts
+    """
     for v in values:
         if not isinstance(v, dict):
             return False
@@ -39,13 +39,11 @@ def is_remote_list(values):
 
 
 def translate_remote(config, setting):
-    """
-        Translate config names from server to equivalents usable
-        in mycroft-core.
+    """Translate config names from server to equivalents for mycroft-core.
 
-        Args:
-                config:     base config to populate
-                settings:   remote settings to be translated
+    Arguments:
+        config:     base config to populate
+        settings:   remote settings to be translated
     """
     IGNORED_SETTINGS = ["uuid", "@type", "active", "user", "device"]
 
@@ -69,12 +67,11 @@ def translate_remote(config, setting):
 
 
 def translate_list(config, values):
-    """
-        Translate list formated by mycroft server.
+    """Translate list formated by mycroft server.
 
-        Args:
-            config (dict): target config
-            values (list): list from mycroft server config
+    Arguments:
+        config (dict): target config
+        values (list): list from mycroft server config
     """
     for v in values:
         module = v["@type"]
@@ -85,9 +82,7 @@ def translate_list(config, values):
 
 
 class LocalConf(dict):
-    """
-        Config dict from file.
-    """
+    """Config dictionary from file."""
     def __init__(self, path):
         super(LocalConf, self).__init__()
         if path:
@@ -95,11 +90,10 @@ def __init__(self, path):
             self.load_local(path)
 
     def load_local(self, path):
-        """
-            Load local json file into self.
+        """Load local json file into self.
 
-            Args:
-                path (str): file to load
+        Arguments:
+            path (str): file to load
         """
         if exists(path) and isfile(path):
             try:
@@ -115,10 +109,10 @@ def load_local(self, path):
             LOG.debug("Configuration '{}' not defined, skipping".format(path))
 
     def store(self, path=None):
-        """
-            Cache the received settings locally. The cache will be used if
-            the remote is unreachable to load settings that are as close
-            to the user's as possible
+        """Cache the received settings locally.
+
+        The cache will be used if the remote is unreachable to load settings
+        that are as close to the user's as possible.
         """
         path = path or self.path
         with open(path, 'w') as f:
@@ -129,9 +123,7 @@ def merge(self, conf):
 
 
 class RemoteConf(LocalConf):
-    """
-        Config dict fetched from mycroft.ai
-    """
+    """Config dictionary fetched from mycroft.ai."""
     def __init__(self, cache=None):
         super(RemoteConf, self).__init__(None)
 
@@ -176,18 +168,23 @@ def __init__(self, cache=None):
 
 
 class Configuration:
+    """Namespace for operations on the configuration singleton."""
     __config = {}  # Cached config
     __patch = {}  # Patch config that skills can update to override config
 
     @staticmethod
     def get(configs=None, cache=True):
-        """
-            Get configuration, returns cached instance if available otherwise
-            builds a new configuration dict.
+        """Get configuration
 
-            Args:
-                configs (list): List of configuration dicts
-                cache (boolean): True if the result should be cached
+        Returns cached instance if available otherwise builds a new
+        configuration dict.
+
+        Arguments:
+            configs (list): List of configuration dicts
+            cache (boolean): True if the result should be cached
+
+        Returns:
+            (dict) configuration dictionary.
         """
         if Configuration.__config:
             return Configuration.__config
@@ -196,14 +193,14 @@ def get(configs=None, cache=True):
 
     @staticmethod
     def load_config_stack(configs=None, cache=False):
-        """
-            load a stack of config dicts into a single dict
+        """Load a stack of config dicts into a single dict
 
-            Args:
-                configs (list): list of dicts to load
-                cache (boolean): True if result should be cached
+        Arguments:
+            configs (list): list of dicts to load
+            cache (boolean): True if result should be cached
 
-            Returns: merged dict of all configuration files
+        Returns:
+            (dict) merged dict of all configuration files
         """
         if not configs:
             configs = [LocalConf(DEFAULT_CONFIG), RemoteConf(),
@@ -233,29 +230,40 @@ def load_config_stack(configs=None, cache=False):
     def set_config_update_handlers(bus):
         """Setup websocket handlers to update config.
 
-        Args:
+        Arguments:
             bus: Message bus client instance
         """
         bus.on("configuration.updated", Configuration.updated)
         bus.on("configuration.patch", Configuration.patch)
+        bus.on("configuration.patch.clear", Configuration.patch_clear)
 
     @staticmethod
     def updated(message):
-        """
-            handler for configuration.updated, triggers an update
-            of cached config.
+        """Handler for configuration.updated,
+
+        Triggers an update of cached config.
         """
         Configuration.load_config_stack(cache=True)
 
     @staticmethod
     def patch(message):
-        """
-            patch the volatile dict usable by skills
+        """Patch the volatile dict usable by skills
 
-            Args:
-                message: Messagebus message should contain a config
-                         in the data payload.
+        Arguments:
+            message: Messagebus message should contain a config
+                     in the data payload.
         """
         config = message.data.get("config", {})
         merge_dict(Configuration.__patch, config)
         Configuration.load_config_stack(cache=True)
+
+    @staticmethod
+    def patch_clear(message):
+        """Clear the config patch space.
+
+        Arguments:
+            message: Messagebus message should contain a config
+                     in the data payload.
+        """
+        Configuration.__patch = {}
+        Configuration.load_config_stack(cache=True)


Test patch:
diff --git a/test/integrationtests/voight_kampff/features/environment.py b/test/integrationtests/voight_kampff/features/environment.py
--- a/test/integrationtests/voight_kampff/features/environment.py
+++ b/test/integrationtests/voight_kampff/features/environment.py
@@ -19,6 +19,7 @@
 
 from msm import MycroftSkillsManager
 from mycroft.audio import wait_while_speaking
+from mycroft.configuration import Configuration
 from mycroft.messagebus.client import MessageBusClient
 from mycroft.messagebus import Message
 from mycroft.util import create_daemon
@@ -92,6 +93,9 @@ def before_all(context):
     context.bus = bus
     context.matched_message = None
     context.log = log
+    context.original_config = {}
+    context.config = Configuration.get()
+    Configuration.set_config_update_handlers(bus)
 
 
 def before_feature(context, feature):
@@ -110,9 +114,26 @@ def after_feature(context, feature):
     sleep(1)
 
 
+def reset_config(context):
+    """Reset configuration with changes stored in original_config of context.
+    """
+    context.log.info('Resetting patched configuration...')
+
+    context.bus.emit(Message('configuration.patch.clear'))
+    key = list(context.original_config)[0]
+    while context.config[key] != context.original_config[key]:
+        sleep(0.5)
+    context.original_config = {}
+
+
 def after_scenario(context, scenario):
+    """Wait for mycroft completion and reset any changed state."""
     # TODO wait for skill handler complete
     sleep(0.5)
     wait_while_speaking()
     context.bus.clear_messages()
     context.matched_message = None
+
+    if context.original_config:
+        # something has changed, reset changes by done in the context
+        reset_config(context)
diff --git a/test/integrationtests/voight_kampff/features/steps/configuration.py b/test/integrationtests/voight_kampff/features/steps/configuration.py
new file mode 100644
--- /dev/null
+++ b/test/integrationtests/voight_kampff/features/steps/configuration.py
@@ -0,0 +1,105 @@
+# Copyright 2020 Mycroft AI Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import json
+from os.path import join, exists
+import time
+
+from behave import given
+from mycroft.messagebus import Message
+from mycroft.util import resolve_resource_file
+
+
+def patch_config(context, patch):
+    """Apply patch to config and wait for it to take effect.
+
+    Arguments:
+        context: Behave context for test
+        patch: patch to apply
+    """
+    # store originals in context
+    for key in patch:
+        # If this patch is redefining an already changed key don't update
+        if key not in context.original_config:
+            context.original_config[key] = context.config.get(key)
+
+    # Patch config
+    patch_config_msg = Message('configuration.patch', {'config': patch})
+    context.bus.emit(patch_config_msg)
+
+    # Wait until one of the keys has been updated
+    key = list(patch.keys())[0]
+    while context.config.get(key) != patch[key]:
+        time.sleep(0.5)
+
+
+def get_config_file_definition(configs_path, config, value):
+    """Read config definition file and return the matching patch dict.
+
+    Arguments:
+        configs_path: path to the configuration patch json file
+        config: config value to fetch from the file
+        value: predefined value to fetch
+
+    Returns:
+        Patch dictionary or None.
+    """
+    with open(configs_path) as f:
+        configs = json.load(f)
+        return configs.get(config, {}).get(value)
+
+
+def get_global_config_definition(context, config, value):
+    """Get config definitions included with Mycroft.
+
+    Arguments:
+        context: behave test context
+        config: config value to fetch from the file
+        value: predefined value to fetch
+
+    Returns:
+        Patch dictionary or None.
+    """
+    configs_path = resolve_resource_file(join('text', context.lang,
+                                              'configurations.json'))
+    return get_config_file_definition(configs_path, config, value)
+
+
+def get_feature_config_definition(context, config, value):
+    """Get config feature specific config defintion
+
+    Arguments:
+        context: behave test context
+        config: config value to fetch from the file
+        value: predefined value to fetch
+
+    Returns:
+        Patch dictionary or None.
+    """
+    feature_config = context.feature.filename.replace('.feature',
+                                                      '.config.json')
+    if exists(feature_config):
+        return get_config_file_definition(feature_config, config, value)
+    else:
+        return None
+
+
+@given('the user\'s {config} is {value}')
+def given_config(context, config, value):
+    """Patch the configuration with a specific config."""
+    config = config.strip('"')
+    value = value.strip('"')
+    patch_dict = (get_feature_config_definition(context, config, value) or
+                  get_global_config_definition(context, config, value))
+    patch_config(context, patch_dict)
diff --git a/test/integrationtests/voight_kampff/features/steps/utterance_responses.py b/test/integrationtests/voight_kampff/features/steps/utterance_responses.py
--- a/test/integrationtests/voight_kampff/features/steps/utterance_responses.py
+++ b/test/integrationtests/voight_kampff/features/steps/utterance_responses.py
@@ -1,4 +1,4 @@
-# Copyright 2017 Mycroft AI Inc.
+# Copyright 2020 Mycroft AI Inc.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/test/integrationtests/voight_kampff/test_setup.py b/test/integrationtests/voight_kampff/test_setup.py
--- a/test/integrationtests/voight_kampff/test_setup.py
+++ b/test/integrationtests/voight_kampff/test_setup.py
@@ -32,12 +32,19 @@
 be found and executed by the behave framework.
 
 The script also ensures that the skills marked for testing are installed and
-that anyi specified extra skills also gets installed into the environment.
+that any specified extra skills also gets installed into the environment.
 """
 
 FEATURE_DIR = join(dirname(__file__), 'features') + '/'
 
 
+def copy_config_definition_files(source, destination):
+    """Copy all feature files from source to destination."""
+    # Copy feature files to the feature directory
+    for f in glob(join(source, '*.config.json')):
+        shutil.copyfile(f, join(destination, basename(f)))
+
+
 def copy_feature_files(source, destination):
     """Copy all feature files from source to destination."""
     # Copy feature files to the feature directory
@@ -141,6 +148,7 @@ def collect_test_cases(msm, skills):
         behave_dir = join(skill.path, 'test', 'behave')
         if exists(behave_dir):
             copy_feature_files(behave_dir, FEATURE_DIR)
+            copy_config_definition_files(behave_dir, FEATURE_DIR)
 
             step_dir = join(behave_dir, 'steps')
             if exists(step_dir):