Repository: qtile/qtile

Issue:
idea: qtile-config {check,migrate}
For a long time we've had issues with configs:

1. people have syntax/logic errors and have to restart a bunch
2. we want to make changes to the internals and people are using them
3. people use the wrong kind of values for config variables, but it works "well enough", except doesn't get the desired behavior
4. etc.

In particular, we've long struggled with number 2, and have no real conclusions about what to support for a 1.0 API. For example, we (rightly so) got rid of a bunch of camel case in our codebase, but it broke everyone's configs who was messing with the internals. I've argued that everything that isn't a `cmd_` is not a public API and we reserve the right to change, but it's still annoying for people to have to update their configs.

Further, r.e. numbers 1 & 3, we've often told people "just run your config via python3 config.py" or (when we get type annotations working) "run pytype on your config". But there are additional checks we could/should have people do (see e.g. #1772) that neither of those will accomplish.

So, what if we introduce a new command `qtile-config`. Initially I'm imagining two subcommands, `check` and `migrate`. `check` would run a type checker, our checks from #1772, and anything else we want (deprecation warnings, etc.). `migrate` would do a best-effort migration of code. In general re-writing code is undecidable, but most of our changes are things like "add another argument to this function" or "rename fooBar to foo_bar", which we could do with reasonable accuracy, and if your config doesn't get auto-migrated, well, too bad.

Then, after 1.0, we allow any API changes where someone can write a migrate blurb that would be reasonably successful at migrating most configs. This means that 1. users don't have do this themselves and 2. we can keep refactoring our code for e.g. wayland.

Once we had this code, we could do other things too, like run `migrate` on qtile start so users' configs are automatically upgraded.

There is an issue of dependencies, but we can make the type checking deps for `qtile-config check` optional, so people don't have to install them if they don't want, and just print a big warning if they're not available.

I'm happy to work on an implementation of this, but I'd just like to see if anyone else has any good ideas before starting :)


Discussion:
I really like the idea! 

We said a few times that we have too many entrypoints, so maybe we should merge all these into qtile beforehand? Something like

```
qtile
qtile help
qtile shell
qtile check
qtile migrate
qtile top
...
```

Also, we could have an automigrate flag in config so that users can optin for us to migrate their config automatically when we detect that qtile version has changed (with the version written in a cache dir or something).

I'll try to think a bit more about this and report back if I have anything else. :)
How that should work in distro packaged? Ex., I do `qtile-git` update in Arch and what should happen?
But I realy like the idea :+1: 
> How that should work in distro packaged? Ex., I do `qtile-git` update in Arch and what should happen?

Well, the user runs the migrate command, or better, you have your package manager doing it for you, or qtile itself does it for you:

> Also, we could have an automigrate flag in config so that users can optin for us to migrate their config automatically when we detect that qtile version has changed (with the version written in a cache dir or something).

At startup we check the version written in the cache, compare it against current version, and migrate if there's a difference.
On Mon, Jun 08, 2020 at 10:41:07PM -0700, Guillaume Gelin wrote:
> I really like the idea! 
> 
> We said a few times that we have too many entrypoints, so maybe we should merge all these into qtile beforehand? Something like
> 
> ```
> qtile
> qtile help
> qtile shell
> qtile check
> qtile migrate
> qtile top
> ...
> ```

That sounds fine, although I think `qtile` should do nothing (and
probably exit(1)), so that people are aware of the new API.

> Also, we could have an automigrate flag in config so that users can optin for us to migrate their config automatically when we detect that qtile version has changed (with the version written in a cache dir or something).

I don't even think we need a version change detection; we can just
require that the transformations in migrate be idempotent.


Resolving patch:
diff --git a/libqtile/scripts/main.py b/libqtile/scripts/main.py
--- a/libqtile/scripts/main.py
+++ b/libqtile/scripts/main.py
@@ -3,7 +3,7 @@
 import sys
 
 from libqtile.log_utils import init_log
-from libqtile.scripts import check, cmd_obj, run_cmd, shell, start, top
+from libqtile.scripts import check, cmd_obj, migrate, run_cmd, shell, start, top
 
 try:
     import pkg_resources
@@ -46,6 +46,7 @@ def main():
     run_cmd.add_subcommand(subparsers)
     cmd_obj.add_subcommand(subparsers)
     check.add_subcommand(subparsers)
+    migrate.add_subcommand(subparsers)
 
     # `qtile help` should print help
     def print_help(options):
diff --git a/libqtile/scripts/migrate.py b/libqtile/scripts/migrate.py
new file mode 100644
--- /dev/null
+++ b/libqtile/scripts/migrate.py
@@ -0,0 +1,116 @@
+# Copyright (c) 2021, Tycho Andersen. All rights reserved.
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+import os
+import os.path
+import shutil
+import sys
+
+BACKUP_SUFFIX = ".migrate.bak"
+
+try:
+    import bowler
+except ImportError:
+    pass
+
+
+def rename_hook(config, fro, to):
+    # could match on dotted_name< 'hook' '.' 'subscribe' '.' '{name}' >
+    # but the replacement gets more complicated...
+    selector = "'{name}'".format(name=fro)
+    q = bowler.Query(config).select_pattern(selector)
+    q.current.kwargs["name"] = fro
+    return q.rename(to)
+
+
+def client_name_updated(config):
+    """ Rename window_name_change -> client_name_updated"""
+    return rename_hook(config, "window_name_change", "client_name_updated")
+
+
+def tile_master_windows_rename(config):
+    return (
+        bowler.Query(config)
+        .select_function("Tile")
+        .modify_argument("masterWindows", "master_length")
+    )
+
+
+def threaded_poll_text_rename(config):
+    return (
+        bowler.Query(config)
+        .select_class("ThreadedPollText")
+        .rename("ThreadPoolText")
+    )
+
+
+MIGRATIONS = [
+    client_name_updated,
+    tile_master_windows_rename,
+    threaded_poll_text_rename,
+]
+
+
+MODULE_RENAMES = [
+    ("libqtile.command_graph", "libqtile.command.graph"),
+    ("libqtile.command_client", "libqtile.command.client"),
+    ("libqtile.command_interface", "libqtile.command.interface"),
+    ("libqtile.command_object", "libqtile.command.object"),
+]
+
+for (fro, to) in MODULE_RENAMES:
+    def f(config, fro=fro, to=to):
+        return (
+            bowler.Query(config)
+            .select_module(fro)
+            .rename(to)
+        )
+    MIGRATIONS.append(f)
+
+
+def do_migrate(args):
+    if "bowler" not in sys.modules:
+        print("bowler can't be found, not migrating config file")
+        print("install it and try again")
+        sys.exit(1)
+
+    shutil.copyfile(args.config, args.config+BACKUP_SUFFIX)
+
+    for m in MIGRATIONS:
+        m(args.config).execute(interactive=args.interactive, write=True)
+
+
+def add_subcommand(subparsers):
+    parser = subparsers.add_parser(
+        "migrate", help="Migrate a configuration file to the current API"
+    )
+    parser.add_argument(
+        "-c",
+        "--config",
+        action="store",
+        default=os.path.expanduser(
+            os.path.join(os.getenv("XDG_CONFIG_HOME", "~/.config"), "qtile", "config.py")
+        ),
+        help="Use the specified configuration file",
+    )
+    parser.add_argument(
+        "--interactive",
+        action="store_true",
+        help="Interactively apply diff (similar to git add -p)",
+    )
+    parser.set_defaults(func=do_migrate)


Test patch:
diff --git a/test/test_migrate.py b/test/test_migrate.py
new file mode 100644
--- /dev/null
+++ b/test/test_migrate.py
@@ -0,0 +1,121 @@
+import hashlib
+import os
+import shutil
+import subprocess
+import tempfile
+import textwrap
+
+from libqtile.scripts.migrate import BACKUP_SUFFIX
+
+
+def run_qtile_migrate(config):
+    cmd = os.path.join(os.path.dirname(__file__), '..', 'bin', 'qtile')
+    argv = [cmd, "migrate", "-c", config]
+    subprocess.check_call(argv)
+
+
+def hash_file(p):
+    with open(p) as f:
+        h = hashlib.sha256()
+        h.update(f.read().encode('utf-8'))
+        return h.digest()
+
+
+def test_migrate_default_config_noop():
+    with tempfile.TemporaryDirectory() as temp:
+        default_config = os.path.join(os.path.dirname(__file__), '..', 'libqtile', 'resources', 'default_config.py')
+        config_path = os.path.join(temp, "config.py")
+        shutil.copyfile(default_config, config_path)
+        run_qtile_migrate(config_path)
+
+        original = hash_file(config_path)
+        migrated = hash_file(config_path+BACKUP_SUFFIX)
+
+        assert original == migrated
+
+
+def read_file(p):
+    with open(p) as f:
+        return f.read()
+
+
+def check_migrate(orig, expected):
+    with tempfile.TemporaryDirectory() as tempdir:
+        config_path = os.path.join(tempdir, "config.py")
+        with open(config_path, "wb") as f:
+            f.write(orig.encode('utf-8'))
+
+        run_qtile_migrate(config_path)
+
+        config_path = os.path.join(tempdir, "config.py")
+        assert expected == read_file(config_path)
+        assert orig == read_file(config_path+BACKUP_SUFFIX)
+
+
+def test_window_name_change():
+    orig = textwrap.dedent("""
+        from libqtile import hook
+
+        @hook.subscribe.window_name_change
+        def f():
+            pass
+    """)
+
+    expected = textwrap.dedent("""
+        from libqtile import hook
+
+        @hook.subscribe.client_name_updated
+        def f():
+            pass
+    """)
+    check_migrate(orig, expected)
+
+
+def test_libqtile_command_graph():
+    orig = textwrap.dedent("""
+        from libqtile.command_graph import CommandObject
+
+        o = CommandObject()
+    """)
+
+    expected = textwrap.dedent("""
+        from libqtile.command.graph import CommandObject
+
+        o = CommandObject()
+    """)
+
+    check_migrate(orig, expected)
+
+
+def test_tile_master_windows():
+    orig = textwrap.dedent("""
+        from libqtile.layout import Tile
+
+        t = Tile(masterWindows=2)
+    """)
+
+    expected = textwrap.dedent("""
+        from libqtile.layout import Tile
+
+        t = Tile(master_length=2)
+    """)
+
+    check_migrate(orig, expected)
+
+
+def test_threaded_poll_text():
+    orig = textwrap.dedent("""
+        from libqtile.widget import ThreadedPollText
+
+        class MyWidget(ThreadedPollText):
+            pass
+    """)
+
+    expected = textwrap.dedent("""
+        from libqtile.widget import ThreadPoolText
+
+        class MyWidget(ThreadPoolText):
+            pass
+    """)
+
+    check_migrate(orig, expected)