Repository: beetbox/beets

Issue:
web: web page search box doesn't work for regex searches
### Problem

This is not a problem in the web API itself, but in the web pages which provide the simple web user interface.

Bringing up the web interface and entering a query such as ``somefield::.`` never returns any results.
The problem is that the web page ends up double URI encoding the search before passing it to GET /item/query.

I have a fix (in `static/beets.js`) which I can submit once the current PR is done.

However, I have no idea how to create a test for this as it would mean starting the webserver, submitting an HTTP request and checking the resulting (complex) HTML. Does anyone have any example of doing that in the beets pytest environment? I know very little python and nothing about pytest but I may be able to steal a similar test if one exists!

EDIT: Actually, it is the last step - parsing and checking the resulting HTML which is hard (the rest is what the tests already do - but they are dealing with JSON responses, not HTML responses). Does anyone have any tools or examples of checking HTML responses? Or do I just do some simple string searches and hope nothing changes too much to change the page in the future?
web: web page search box doesn't work for regex searches
### Problem

This is not a problem in the web API itself, but in the web pages which provide the simple web user interface.

Bringing up the web interface and entering a query such as ``somefield::.`` never returns any results.
The problem is that the web page ends up double URI encoding the search before passing it to GET /item/query.

I have a fix (in `static/beets.js`) which I can submit once the current PR is done.

However, I have no idea how to create a test for this as it would mean starting the webserver, submitting an HTTP request and checking the resulting (complex) HTML. Does anyone have any example of doing that in the beets pytest environment? I know very little python and nothing about pytest but I may be able to steal a similar test if one exists!

EDIT: Actually, it is the last step - parsing and checking the resulting HTML which is hard (the rest is what the tests already do - but they are dealing with JSON responses, not HTML responses). Does anyone have any tools or examples of checking HTML responses? Or do I just do some simple string searches and hope nothing changes too much to change the page in the future?


Discussion:
Good catch; that seems like a good thing to fix!

My honest feeling here is that this is probably not worth writing a test for. I think it would be pretty annoying to test a combination of JavaScript and Python, and especially so to test something actually running in a browser. Tests are nice to have, certainly, but in some circumstances they are just not worth the hassle. :smiley:
Thanks for the advice Adrian. I will be including a couple of tests for the underlying (JSON) web API operations. These currently work but it doesn't do any harm to have sometests to make sure they don't get broken in the future.

Unless I can think of an easy (and reliable) way to add a test for the JS code I will leave it out. Tests which are flaky, or break due to harmless other changes, tend to make things harder instead of easier!
This has got a bit more complex. There are two related but different bugs, both in `web/__init__.py` class `QueryConverter`:

1. Regex's which include `\` (e.g. `\d` or `match a square bracket \[`) fall foul of special code which converts `\` to the os fileseparator. 
2. Any query which includes `/` fails because the class joins `/` separated parts of the URL together (dropping the `/`), but it does it **after** URL decode so there is no way to ever preserve a `/`.

I have a simple fix which just does not do replacement 1 if the query also includes `:` (e.g. `field:value` or `field::value`).  This works to make regexs using `\` work (which is critical if we are going to make regex useful). But it doesn't fix using `/` or using `\` outside field matches.

To be honest, I am not convinced of the value of doing replacement 1 at all. It seems to be intended to allow path specifications to always use `\`, both for portability between OS and to avoid interacting with replacement 2. However, _queries_ trying to match paths (even without any slashes) don't seem to work anyway (even with `include_paths` set) - with or without my change. Is there some circumstance where being able to use a filespec in a `/item/query` operation (as opposed to a `/item/path` operation) is useful?

A possibly related question is that explicitly trying to query on paths doesn't seem to work very well _anywhere_ in beets! At the command line, `beet ls path::Edition` displays all my tracks which have Edition in the path. And `beet ls path::Edition/01` displays the files which have a directory ending in Edition and a filename beginning 01, as expected.  However neither `beet ls path:Edition` nor `beet ls path:Edition/01` (i.e. without regex) display anything. Should they?

Anyway, I am not trying to open up a larger can of worms. But I am trying to work out whether to include my limited fix to stop replacement 1 in queries including `:` (or limit it only to regex queries (`::`)) or remove replacement 1 (and replacement 2?) altogether. My fix seems to allow the cases I care about (using useful regex matches for non-path fields), and avoids potential wider damage, so I will propose that unless someone cares enough to suggest something different.
Wow, that is pretty nasty! I think you're right about the reason why replacement 1 exists, and the original idea to allow queries like `foo:bar/baz:qux` to encode two query parts was probably an overly RESTful mistake. It might be useful to dig into the git history to see if we can uncover some history about why that was added—presumably somebody had a justification for why they wanted to be able to do path queries and couldn't without this change?

> However neither `beet ls path:Edition` nor `beet ls path:Edition/01` (i.e. without regex) display anything. Should they?

No—path queries get handled specially so they query _entire directory names_ instead of substring matches. So `path:Edition/01` finds everything in your library that's inside a directory `./Edition/01/`. Here's where that happens in the source:
https://github.com/beetbox/beets/blob/3e8261393d57e33edd7e7956b71245f3d249d7c7/beets/library.py#L53

And here's the somewhat high-level documentation:
https://beets.readthedocs.io/en/stable/reference/query.html#path-queries
Thanks for the hint. I checked the git log for `web/__init__.py` and discovered the background to the code. Basically it is described in issue #3566 and https://github.com/beetbox/beets/commit/6a03afc65df168cf60b83bc34c1b9b94c648c629  (@nmeum )

My proposed change (to stop doing the \ substitution if a field match is happening - i.e. a `:` is present) would explicitly break the example in #3566, which is a `path:` match. So that is not going to be acceptable.

I suspect the real fix is to undo both replacements. My guess is that there are a tiny number of people using either feature and they could easily change to a world without them. But that might mean code changes for existing users so is not desirable.

My new proposal is to change replacement 1 so it does not apply to **regex** queries (which are much more likely to use `\` than normal queries). In other words, do not do it if `::` is in the query. Although this is a change, and will break regex queries for paths, I think it is very unlikely anyone is doing regex queries on paths today.

An alternative is to deprecate but maintain the current syntax and introduce a new query format which does not do manipulation on `/` or `\`. But that seems a bit pointless as AURA is coming (although I haven't checked whether it supports these sorts of queries).

It would be good to hear from @nmeum if they are still around.
Great. Changing things just for queries that contain `::` sounds like a good near-term/immediate fix; then, we can consider rethinking this more carefully in the future. (Or just switching over to the new AURA server plugin from #3758 instead…)
Good catch; that seems like a good thing to fix!

My honest feeling here is that this is probably not worth writing a test for. I think it would be pretty annoying to test a combination of JavaScript and Python, and especially so to test something actually running in a browser. Tests are nice to have, certainly, but in some circumstances they are just not worth the hassle. :smiley:
Thanks for the advice Adrian. I will be including a couple of tests for the underlying (JSON) web API operations. These currently work but it doesn't do any harm to have sometests to make sure they don't get broken in the future.

Unless I can think of an easy (and reliable) way to add a test for the JS code I will leave it out. Tests which are flaky, or break due to harmless other changes, tend to make things harder instead of easier!
This has got a bit more complex. There are two related but different bugs, both in `web/__init__.py` class `QueryConverter`:

1. Regex's which include `\` (e.g. `\d` or `match a square bracket \[`) fall foul of special code which converts `\` to the os fileseparator. 
2. Any query which includes `/` fails because the class joins `/` separated parts of the URL together (dropping the `/`), but it does it **after** URL decode so there is no way to ever preserve a `/`.

I have a simple fix which just does not do replacement 1 if the query also includes `:` (e.g. `field:value` or `field::value`).  This works to make regexs using `\` work (which is critical if we are going to make regex useful). But it doesn't fix using `/` or using `\` outside field matches.

To be honest, I am not convinced of the value of doing replacement 1 at all. It seems to be intended to allow path specifications to always use `\`, both for portability between OS and to avoid interacting with replacement 2. However, _queries_ trying to match paths (even without any slashes) don't seem to work anyway (even with `include_paths` set) - with or without my change. Is there some circumstance where being able to use a filespec in a `/item/query` operation (as opposed to a `/item/path` operation) is useful?

A possibly related question is that explicitly trying to query on paths doesn't seem to work very well _anywhere_ in beets! At the command line, `beet ls path::Edition` displays all my tracks which have Edition in the path. And `beet ls path::Edition/01` displays the files which have a directory ending in Edition and a filename beginning 01, as expected.  However neither `beet ls path:Edition` nor `beet ls path:Edition/01` (i.e. without regex) display anything. Should they?

Anyway, I am not trying to open up a larger can of worms. But I am trying to work out whether to include my limited fix to stop replacement 1 in queries including `:` (or limit it only to regex queries (`::`)) or remove replacement 1 (and replacement 2?) altogether. My fix seems to allow the cases I care about (using useful regex matches for non-path fields), and avoids potential wider damage, so I will propose that unless someone cares enough to suggest something different.
Wow, that is pretty nasty! I think you're right about the reason why replacement 1 exists, and the original idea to allow queries like `foo:bar/baz:qux` to encode two query parts was probably an overly RESTful mistake. It might be useful to dig into the git history to see if we can uncover some history about why that was added—presumably somebody had a justification for why they wanted to be able to do path queries and couldn't without this change?

> However neither `beet ls path:Edition` nor `beet ls path:Edition/01` (i.e. without regex) display anything. Should they?

No—path queries get handled specially so they query _entire directory names_ instead of substring matches. So `path:Edition/01` finds everything in your library that's inside a directory `./Edition/01/`. Here's where that happens in the source:
https://github.com/beetbox/beets/blob/3e8261393d57e33edd7e7956b71245f3d249d7c7/beets/library.py#L53

And here's the somewhat high-level documentation:
https://beets.readthedocs.io/en/stable/reference/query.html#path-queries
Thanks for the hint. I checked the git log for `web/__init__.py` and discovered the background to the code. Basically it is described in issue #3566 and https://github.com/beetbox/beets/commit/6a03afc65df168cf60b83bc34c1b9b94c648c629  (@nmeum )

My proposed change (to stop doing the \ substitution if a field match is happening - i.e. a `:` is present) would explicitly break the example in #3566, which is a `path:` match. So that is not going to be acceptable.

I suspect the real fix is to undo both replacements. My guess is that there are a tiny number of people using either feature and they could easily change to a world without them. But that might mean code changes for existing users so is not desirable.

My new proposal is to change replacement 1 so it does not apply to **regex** queries (which are much more likely to use `\` than normal queries). In other words, do not do it if `::` is in the query. Although this is a change, and will break regex queries for paths, I think it is very unlikely anyone is doing regex queries on paths today.

An alternative is to deprecate but maintain the current syntax and introduce a new query format which does not do manipulation on `/` or `\`. But that seems a bit pointless as AURA is coming (although I haven't checked whether it supports these sorts of queries).

It would be good to hear from @nmeum if they are still around.
Great. Changing things just for queries that contain `::` sounds like a good near-term/immediate fix; then, we can consider rethinking this more carefully in the future. (Or just switching over to the new AURA server plugin from #3758 instead…)

Resolving patch:
diff --git a/beetsplug/web/__init__.py b/beetsplug/web/__init__.py
--- a/beetsplug/web/__init__.py
+++ b/beetsplug/web/__init__.py
@@ -244,7 +244,9 @@ class QueryConverter(PathConverter):
 
     def to_python(self, value):
         queries = value.split('/')
-        return [query.replace('\\', os.sep) for query in queries]
+        """Do not do path substitution on regex value tests"""
+        return [query if '::' in query else query.replace('\\', os.sep)
+                for query in queries]
 
     def to_url(self, value):
         return ','.join([v.replace(os.sep, '\\') for v in value])


Test patch:
diff --git a/test/test_web.py b/test/test_web.py
--- a/test/test_web.py
+++ b/test/test_web.py
@@ -13,11 +13,22 @@
 from beets.library import Item, Album
 from beetsplug import web
 
+import platform
+
+from beets import logging
+
 
 class WebPluginTest(_common.LibTestCase):
 
     def setUp(self):
+
         super(WebPluginTest, self).setUp()
+        self.log = logging.getLogger('beets.web')
+
+        if platform.system() == 'Windows':
+            self.path_prefix = u'C:'
+        else:
+            self.path_prefix = u''
 
         # Add fixtures
         for track in self.lib.items():
@@ -26,12 +37,30 @@ def setUp(self):
         # Add library elements. Note that self.lib.add overrides any "id=<n>"
         # and assigns the next free id number.
         # The following adds will create items #1, #2 and #3
-        self.lib.add(Item(title=u'title', path='/path_1', album_id=2))
-        self.lib.add(Item(title=u'another title', path='/path_2'))
-        self.lib.add(Item(title=u'and a third'))
+        path1 = self.path_prefix + os.sep + \
+            os.path.join(b'path_1').decode('utf-8')
+        self.lib.add(Item(title=u'title',
+                          path=path1,
+                          album_id=2,
+                          artist='AAA Singers'))
+        path2 = self.path_prefix + os.sep + \
+            os.path.join(b'somewhere', b'a').decode('utf-8')
+        self.lib.add(Item(title=u'another title',
+                          path=path2,
+                          artist='AAA Singers'))
+        path3 = self.path_prefix + os.sep + \
+            os.path.join(b'somewhere', b'abc').decode('utf-8')
+        self.lib.add(Item(title=u'and a third',
+                          testattr='ABC',
+                          path=path3,
+                          album_id=2))
         # The following adds will create albums #1 and #2
-        self.lib.add(Album(album=u'album'))
-        self.lib.add(Album(album=u'other album', artpath='/art_path_2'))
+        self.lib.add(Album(album=u'album',
+                           albumtest='xyz'))
+        path4 = self.path_prefix + os.sep + \
+            os.path.join(b'somewhere2', b'art_path_2').decode('utf-8')
+        self.lib.add(Album(album=u'other album',
+                           artpath=path4))
 
         web.app.config['TESTING'] = True
         web.app.config['lib'] = self.lib
@@ -42,17 +71,25 @@ def test_config_include_paths_true(self):
         web.app.config['INCLUDE_PATHS'] = True
         response = self.client.get('/item/1')
         res_json = json.loads(response.data.decode('utf-8'))
+        expected_path = self.path_prefix + os.sep \
+            + os.path.join(b'path_1').decode('utf-8')
 
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(res_json['path'], u'/path_1')
+        self.assertEqual(res_json['path'], expected_path)
+
+        web.app.config['INCLUDE_PATHS'] = False
 
     def test_config_include_artpaths_true(self):
         web.app.config['INCLUDE_PATHS'] = True
         response = self.client.get('/album/2')
         res_json = json.loads(response.data.decode('utf-8'))
+        expected_path = self.path_prefix + os.sep \
+            + os.path.join(b'somewhere2', b'art_path_2').decode('utf-8')
 
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(res_json['artpath'], u'/art_path_2')
+        self.assertEqual(res_json['artpath'], expected_path)
+
+        web.app.config['INCLUDE_PATHS'] = False
 
     def test_config_include_paths_false(self):
         web.app.config['INCLUDE_PATHS'] = False
@@ -91,8 +128,8 @@ def test_get_multiple_items_by_id(self):
 
         self.assertEqual(response.status_code, 200)
         self.assertEqual(len(res_json['items']), 2)
-        response_titles = [item['title'] for item in res_json['items']]
-        assertCountEqual(self, response_titles, [u'title', u'another title'])
+        response_titles = {item['title'] for item in res_json['items']}
+        self.assertEqual(response_titles, {u'title', u'another title'})
 
     def test_get_single_item_not_found(self):
         response = self.client.get('/item/4')
@@ -116,6 +153,7 @@ def test_get_single_item_by_path_not_found_if_not_in_library(self):
         self.assertEqual(response.status_code, 404)
 
     def test_get_item_empty_query(self):
+        """ testing item query: <empty> """
         response = self.client.get('/item/query/')
         res_json = json.loads(response.data.decode('utf-8'))
 
@@ -123,6 +161,7 @@ def test_get_item_empty_query(self):
         self.assertEqual(len(res_json['items']), 3)
 
     def test_get_simple_item_query(self):
+        """ testing item query: another """
         response = self.client.get('/item/query/another')
         res_json = json.loads(response.data.decode('utf-8'))
 
@@ -131,6 +170,52 @@ def test_get_simple_item_query(self):
         self.assertEqual(res_json['results'][0]['title'],
                          u'another title')
 
+    def test_query_item_string(self):
+        """ testing item query: testattr:ABC """
+        response = self.client.get('/item/query/testattr%3aABC')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['title'],
+                         u'and a third')
+
+    def test_query_item_regex(self):
+        """ testing item query: testattr::[A-C]+ """
+        response = self.client.get('/item/query/testattr%3a%3a[A-C]%2b')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['title'],
+                         u'and a third')
+
+    def test_query_item_regex_backslash(self):
+        # """ testing item query: testattr::\w+ """
+        response = self.client.get('/item/query/testattr%3a%3a%5cw%2b')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['title'],
+                         u'and a third')
+
+    def test_query_item_path(self):
+        # """ testing item query: path:\somewhere\a """
+        """ Note: path queries are special: the query item must match the path
+        from the root all the way to a directory, so this matches 1 item """
+        """ Note: filesystem separators in the query must be '\' """
+
+        response = self.client.get('/item/query/path:'
+                                   + self.path_prefix
+                                   + '\\somewhere\\a')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['title'],
+                         u'another title')
+
     def test_get_all_albums(self):
         response = self.client.get('/album/')
         res_json = json.loads(response.data.decode('utf-8'))
@@ -177,10 +262,43 @@ def test_get_album_details(self):
         res_json = json.loads(response.data.decode('utf-8'))
 
         self.assertEqual(response.status_code, 200)
-        self.assertEqual(len(res_json['items']), 1)
+        self.assertEqual(len(res_json['items']), 2)
         self.assertEqual(res_json['items'][0]['album'],
                          u'other album')
-        self.assertEqual(res_json['items'][0]['id'], 1)
+        self.assertEqual(res_json['items'][1]['album'],
+                         u'other album')
+        response_track_titles = {item['title'] for item in res_json['items']}
+        self.assertEqual(response_track_titles, {u'title', u'and a third'})
+
+    def test_query_album_string(self):
+        """ testing query: albumtest:xy """
+        response = self.client.get('/album/query/albumtest%3axy')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['album'],
+                         u'album')
+
+    def test_query_album_artpath_regex(self):
+        """ testing query: artpath::art_ """
+        response = self.client.get('/album/query/artpath%3a%3aart_')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['album'],
+                         u'other album')
+
+    def test_query_album_regex_backslash(self):
+        # """ testing query: albumtest::\w+ """
+        response = self.client.get('/album/query/albumtest%3a%3a%5cw%2b')
+        res_json = json.loads(response.data.decode('utf-8'))
+
+        self.assertEqual(response.status_code, 200)
+        self.assertEqual(len(res_json['results']), 1)
+        self.assertEqual(res_json['results'][0]['album'],
+                         u'album')
 
     def test_get_stats(self):
         response = self.client.get('/stats')