Repository: talonhub/community

Issue:
lists/dicts in vocabulary in separate json files for easier merging of changes to vocabulary.py
As it stands, many refactors to `vocabulary.py` result in merge conflicts due to custom vocabulary defined. If these were instead read from json files, this would no longer be a problem.


Discussion:
good idea. I think we should be able to do something like what homophones.py does, and load them from csv files (even simpler than json :P)
Yeah, CSV works and has precedent 👍 I think jcaw might have some work in this direction as well
Specifically, something standard would be best. CSV, Json, etc. That way people dont have to learn some new format, how do you escape commas? How do you use quotes? What about commas inside quotes? Just pick a standard and stick to it rather than invent yet another file format
Also a reminder of `resource.open` which allows talon to do auto reloading on file changes.
CSVs have the bonus that we can parse them line-by-line and ignore bad lines, whereas a JSON parse error will bring down the whole file.

I think if we do use CSV, we'll need to lay it out with headers:

```
| Word | Substitution (Optional) |
| ---- | ----------------------- |
| cash | cache                   |
| def  |                         |  <-- This will map to itself, "def": "def"
```

But the header could be a point of confusion unless formatting is baked in. 

```
Word,Substition
cash,cache
```

This could be a generic format that allows the user to define Talon list contents in general, until there's a first-party solution. The only problem with CSV (and JSON) is that it doesn't support comments.

How would you want to handle words that need to be capitalized? 
- We could force the user to write out both forms
- We could have another CSV that just has one column, but that breaks the pattern of `spoken_form: optional_substitute`. 
- We could include the capitalized words in one column and perform some magic to map them when they're added to the substitition list. This seems confusing though.
Hmm... What about this?

Additional words:

```
| Word(s) | Spoken Form (If Different) |
| ------- | -------------------------- |
| nmap    | enn map                    |
| def     |                            |
```

Then for replacements, including capitalized:

```
| Replacement | Original (Optional) |
| ----------- | ------------------- |
| cache       | cash                |
| December    |                     |
```
And for multiples:

```
| Word(s) | Spoken Form (If Different) |
| ------- | -------------------------- |
| nmap    |                            |
| nmap    | enn map                    |
| nmap    | N map                      |
```

Since the key is on the right, to maintain dict semantics we can just error out on rows where a non-empty spoken form is redefined.
multiple files sound good to me, essentially replicating what is currently in vocabulary.py. Just to be clear though, I think we should use CSV format for now though, the header is okay:

```
output, spoken form
cache, cash
nmap, N map
nmap, enn map
```

this all looks good to me, will definitely make it easier to manage. I think using csv format is fine
I've got an implementation working using the file format I described here, so it's no extra work to use that (probably less tbh, since it's just one format). I can do either approach.

@knausj85 @rntz (and anyone else), thoughts on preference? Two files in the structure above, or four files replicating the current structure?
Another benefit of doing it that way is the system could be used anywhere we need to define a Talon list in a user file, and they'd all have the same basic format.
@jcaw I'm confused about your proposal. Your examples have a lot of formatting in them (`|` and `-`) – do you propose including that in the actual files, or is that merely to make the examples clearer? I would prefer a format closer to csv, pseudo-table formatting seems unnecessary and difficult to maintain over a large file.

My main concerns with csv are support for comments and dealing with leading and trailing white space or other special characters. Comments are easy enough to add, we can just use `#` at the beginning of the line. White space and special characters aren't an issue with any of our current use cases, so I think we can punt on that for now and add it later unless someone specifically wants to deal with it.

Is there a reason not to just use comments as headers?

I have no strong preference regarding having two files versus four files,  or whether the key/spoken form or the value/written form come first.
I just drew the structure for visualisation, the format is a normal CSV:

```
Header 1,Header 2
value 1,key 1
value 2
value 3,key 3
```

Whitespace and special characters should work fine with CSV (edit: but it might require playing with [encoding](https://stackoverflow.com/a/37364412/3255378) a bit). Escaping is easy, you just use quotes: `normal value,"escaped, value"`. You only need to escape commas and quotes - padding is respected.

I'm against modifying vanilla CSVs to add comments. It's too unexpected. I think something like that would require an explicitly new format.
sounds good to me then, if somebody makes a PR for this I'd be happy to review/accept it :)

In fact, if we're going to be using vanilla csv I think there's a python module for that in the standard library: https://docs.python.org/3/library/csv.html
sounds like everyone's on the same page, but FWIW I'd vote to keeping it simple & clean- 

(1) Use CSV; probably stick with vanilla CSV stuff since this is (presumably) a temporary solution until .talon has support for lists. Just document when/what we can.
(2) 4 files is probably the most intuitive for newbies, particularly with the differences between Dragon and w2l re: capitalization. 

Resolving patch:
diff --git a/code/user_settings.py b/code/user_settings.py
new file mode 100644
--- /dev/null
+++ b/code/user_settings.py
@@ -0,0 +1,145 @@
+from talon import Module, fs, Context
+import os
+import csv
+from pathlib import Path
+from typing import Dict, List, Tuple
+import threading
+
+
+# NOTE: This method requires this module to be one folder below the top-level
+#   knausj folder.
+SETTINGS_DIR = Path(__file__).parents[1] / "settings"
+
+
+mod = Module()
+ctx = Context()
+
+
+def _load_csv_dict(
+    file_name: str, headers=Tuple[str, str], default: Dict[str, str] = {}
+) -> Dict[str, str]:
+    """Load a word mapping from a CSV file. If it doesn't exist, create it."""
+    assert file_name.endswith(".csv")
+    path = SETTINGS_DIR / file_name
+
+    # Create the file if it doesn't exist
+    if not SETTINGS_DIR.is_dir():
+        os.mkdir(SETTINGS_DIR)
+    if not path.is_file():
+        with open(path, "w", encoding="utf-8") as file:
+            writer = csv.writer(file)
+            writer.writerow(headers)
+            for key, value in default.items():
+                writer.writerow([key] if key == value else [value, key])
+
+    # Now read from disk
+    with open(path, "r", encoding="utf-8") as file:
+        rows = list(csv.reader(file))
+
+    mapping = {}
+    if len(rows) >= 2:
+        actual_headers = rows[0]
+        if not actual_headers == list(headers):
+            print(
+                f'"{file_name}": Malformed headers - {actual_headers}.'
+                + f" Should be {list(headers)}. Ignoring row."
+            )
+        for row in rows[1:]:
+            if len(row) == 0:
+                # Windows newlines are sometimes read as empty rows. :champagne:
+                continue
+            if len(row) == 1:
+                mapping[row[0]] = row[0]
+            else:
+                mapping[row[1]] = row[0]
+                if len(row) > 2:
+                    print(
+                        f'"{file_name}": More than two values in row: {row}.'
+                        + " Ignoring the extras."
+                    )
+    return mapping
+
+
+_mapped_lists = {}
+_settings_lock = threading.Lock()
+_word_map_params = None
+
+
+def _update_list(list_name: str, *csv_params):
+    """Update list with `list_name` from a csv on disk.
+
+    `csv_params` will be passed to `_load_csv_dict`.
+
+    """
+    global ctx
+    ctx.lists[list_name] = _load_csv_dict(*csv_params)
+
+
+def _update_word_map(*csv_params):
+    """Update `dictate.word_map` from disk.
+
+    `csv_params` will be passed to `_load_csv_dict`.
+
+    """
+    global ctx
+    ctx.settings["dictate.word_map"] = _load_csv_dict(*csv_params)
+
+
+def _update_lists(*_):
+    """Update all CSV lists from disk."""
+    print("Updating CSV lists...")
+    with _settings_lock:
+        for list_name, csv_params in _mapped_lists.items():
+            try:
+                _update_list(list_name, *csv_params)
+            except Exception as e:
+                print(f'Error loading list "{list_name}": {e}')
+        # Special case - `dictate.word_map` isn't a list.
+        if _word_map_params:
+            try:
+                _update_word_map(*_word_map_params)
+            except Exception as e:
+                print(f'Error updating "dictate.word_map": {e}')
+
+
+def bind_list_to_csv(
+    list_name: str,
+    csv_name: str,
+    csv_headers: Tuple[str, str],
+    default_values: Dict[str, str] = {},
+) -> None:
+    """Register a Talon list that should be updated from a CSV on disk.
+
+    The CSV file will be created automatically in the "settings" dir if it
+    doesn't exist. This directory can be tracked independently to
+    `knausj_talon`, allowing the user to specify things like private vocab
+    separately.
+
+    Note the list must be declared separately.
+
+    """
+    global _mapped_lists
+    with _settings_lock:
+        _update_list(list_name, csv_name, csv_headers, default_values)
+        # If there were no errors, we can register it permanently.
+        _mapped_lists[list_name] = (csv_name, csv_headers, default_values)
+
+
+def bind_word_map_to_csv(
+    csv_name: str, csv_headers: Tuple[str, str], default_values: Dict[str, str] = {}
+) -> None:
+    """Like `bind_list_to_csv`, but for the `dictate.word_map` setting.
+
+    Since it is a setting, not a list, it has to be handled separately.
+
+    """
+    global _word_map_params
+    # TODO: Maybe a generic system for binding the dicts to settings? Only
+    #   implement if it's needed.
+    with _settings_lock:
+        _update_word_map(csv_name, csv_headers, default_values)
+        # If there were no errors, we can register it permanently.
+        _word_map_params = (csv_name, csv_headers, default_values)
+
+
+fs.watch(str(SETTINGS_DIR), _update_lists)
diff --git a/code/vocabulary.py b/code/vocabulary.py
--- a/code/vocabulary.py
+++ b/code/vocabulary.py
@@ -1,8 +1,9 @@
 from talon import Context, Module, actions, grammar
+from .user_settings import bind_list_to_csv, bind_word_map_to_csv
 
-# Add single words here if Talon recognizes them, but they need to have their
-# capitalization adjusted.
-capitalize = [
+
+# Default words that will need to be capitalized (particularly under w2l).
+_capitalize_defaults = [
     "I",
     "I'm",
     "I've",
@@ -36,26 +37,21 @@
     "December",
 ]
 
-# Add single words here if Talon recognizes them, but they need to have their
-# spelling adjusted.
-word_map = {
-    # For example:
-    # "color": "colour",
+# Default words that need to be remapped.
+_word_map_defaults = {
+    # E.g:
+    # "cash": "cache",
 }
-word_map.update({x.lower(): x for x in capitalize})
-
-# Add words (or phrases you want treated as words) here if Talon doesn't
-# recognize them at all.
-simple_vocabulary = ["nmap", "admin", "Cisco", "Citrix", "VPN", "DNS", "minecraft"]
-
-# Add vocabulary words (or phrases you want treated as words) here that aren't
-# recognized by Talon and are written differently than they're pronounced.
-mapping_vocabulary = {
-    # For example:
-    # "enn map": "nmap",
-    # "under documented": "under-documented",
+
+# Default words that should be added to Talon's vocabulary.
+_simple_vocab_default = ["nmap", "admin", "Cisco", "Citrix", "VPN", "DNS", "Minecraft"]
+
+# Defaults for different pronounciations of words that need to be added to
+# Talon's vocabulary.
+_mapping_vocab_default = {
+    "N map": "nmap",
+    "under documented": "under-documented",
 }
-mapping_vocabulary.update(dict(zip(simple_vocabulary, simple_vocabulary)))
 
 
 mod = Module()
@@ -106,12 +102,28 @@ def text(m) -> str:
 
 ctx = Context()
 
-# dictate.word_map is used by actions.dictate.replace_words to rewrite words
+
+_default_word_map = {}
+_default_word_map.update(_word_map_defaults)
+_default_word_map.update({word.lower(): word for word in _capitalize_defaults})
+# "dictate.word_map" is used by `actions.dictate.replace_words` to rewrite words
 # Talon recognized. Entries in word_map don't change the priority with which
 # Talon recognizes some words over others.
-ctx.settings["dictate.word_map"] = word_map
-
-# user.vocabulary is used to explicitly add words/phrases that Talon doesn't
+bind_word_map_to_csv(
+    "words_to_replace.csv",
+    csv_headers=("Replacement", "Original"),
+    default_values=_default_word_map,
+)
+
+_default_vocabulary = {}
+_default_vocabulary.update({word: word for word in _simple_vocab_default})
+_default_vocabulary.update(_mapping_vocab_default)
+# "user.vocabulary" is used to explicitly add words/phrases that Talon doesn't
 # recognize. Words in user.vocabulary (or other lists and captures) are
 # "command-like" and their recognition is prioritized over ordinary words.
-ctx.lists["user.vocabulary"] = mapping_vocabulary
+bind_list_to_csv(
+    "user.vocabulary",
+    "additional_words.csv",
+    csv_headers=("Word(s)", "Spoken Form (If Different)"),
+    default_values=_default_vocabulary,
+)


Test patch: