Repository: beetbox/beets

Issue:
Case insensitive search for a file does not work
Imagine you have a file at `/path/to/library/A/File.mp3`. Executing `beet ls -f $path` will give us

```
/path/to/library/A/File.mp3
```

So I would expect to get a result when executing `beet ls path:"/path/to/library/A/File.mp3"`, but beets doesn't find the file.

I tried to find the bug but didn't succeed. One thing I noted is the where clause of the `PathQuery` (line 103 of `library.py`): The bound parameter is in lower case but the data at the database might contain upper case letters. Therefore, a check like `path = ?` might fail even if it shouldn't. So I changed it to `lower(path) = ?` but that didn't change the described behavior.

What am I doing wrong?

Case insensitive search for a file does not work
Imagine you have a file at `/path/to/library/A/File.mp3`. Executing `beet ls -f $path` will give us

```
/path/to/library/A/File.mp3
```

So I would expect to get a result when executing `beet ls path:"/path/to/library/A/File.mp3"`, but beets doesn't find the file.

I tried to find the bug but didn't succeed. One thing I noted is the where clause of the `PathQuery` (line 103 of `library.py`): The bound parameter is in lower case but the data at the database might contain upper case letters. Therefore, a check like `path = ?` might fail even if it shouldn't. So I changed it to `lower(path) = ?` but that didn't change the described behavior.

What am I doing wrong?



Discussion:
That's odd—I can't reproduce this on either of my main systems (one OS X, one Arch Linux). Typing `beet ls path:` and the path to a file works fine. My OS X disk is case-insensitive and the Linux filesystem is of course case-sensitive.

Any idea what might be different about your system?

I can attest that sometimes it seemed to me that `beet ls path:<path>` would behave weirdly, but I always thought  the problem was that I did not enter the path correctly from a bash point of view (protecting with quotes, escaping characters, etc). So every time it happened, I ended up changing the syntax a bunch of times until it worked eventually (and surprisingly). But again, I always thought this was coming from me not being able to remember the syntax accurately when it comes to long files with a lot of unicode gibberish, not from beets.

Maybe I'll try to reproduce this and then see if something pops up.

I also tried to reproduced it, but without success.

To avoid any problems with escaping characters, quotes, etc.: Should we introduced an output option to the ls command which will print the query at first and then prints the results (and a summary maybe)

```
beet ls path:"/path/to/library/A/File.mp3"
```

could give an output like

```
Searching for:
path:"/path/to/library/A/File.mp3"

Results:
Beatles - Let it be - Let it be

Summary:
Found 1 track
```

What do you think?

Some additional debugging output might serve this role well too.

I too think it could be a useful debug info to have but that it indeed wouldn't belong to every day output.

After digging a bit more into the code I found something which might help to understand what's going on here. I guess the root of the problem is the type of the `path`-column at the database: It's a BLOB. Using a BLOB column within a where clause can be tricky sometimes.
I've changed the where clause from

```
({0} = ?) || ({0} LIKE ? ESCAPE ?)
```

to

```
(lower({0}) = lower(?)) || (lower({0}) LIKE lower(?) ESCAPE ?)
```

and now it seems to work for me. But I'm really unsure if this is the right solution.

Any comments?

Aha! I think so! In the case-insensitive case (sorry), the pattern is lower-cased: https://github.com/sampsyo/beets/blob/master/beets/library.py#L71

So checking equality as in `{0} = ?` isn't right; we also need to lower-case the value from the database.

Does just changing it to this work?

```
(lower({0}) = ?) || ({0} LIKE ? ESCAPE ?)
```

That is, the pattern is already lower-cased, so no need for `lower(?)`. And LIKE is already case-insensitive, so leave that alone. I think we just need `lower` on the value in the equality.

No, that doesn't do the trick. But you are right with the `LIKE` part of the where clause: we don't need the `lower` there. So the following works perfect for me (at least with the test cases I've got here...):

```
(lower({0}) = lower(?)) || ({0} LIKE ? ESCAPE ?)
```

But I'm still not sure why we have to do a `lower` for the parameter since the string is already in lower case. Maybe Python and SQLite create something different when using `lower`?

Good thinking. Do you think you might be able to use print-debugging to track down a case where Python's `path.lower()` doesn't match SQLite's `lower(path)`? Then we can really nail this bug.

In any case, if there really is a difference, then perhaps we should keep `LOWER({0}) = LOWER(?)` in that query and then _remove_ the lower-casing on the Python side.

I did as you suggested: Selected `lower(path)` and used `path.lower()` in Python and printed the byte representation of each character. They are both absolutely identical.

Removing the lower-casing on the Python side seems to have no effect, hence everything is still working perfectly. I'll prepare a PR containing these changes.

That's odd—I can't reproduce this on either of my main systems (one OS X, one Arch Linux). Typing `beet ls path:` and the path to a file works fine. My OS X disk is case-insensitive and the Linux filesystem is of course case-sensitive.

Any idea what might be different about your system?

I can attest that sometimes it seemed to me that `beet ls path:<path>` would behave weirdly, but I always thought  the problem was that I did not enter the path correctly from a bash point of view (protecting with quotes, escaping characters, etc). So every time it happened, I ended up changing the syntax a bunch of times until it worked eventually (and surprisingly). But again, I always thought this was coming from me not being able to remember the syntax accurately when it comes to long files with a lot of unicode gibberish, not from beets.

Maybe I'll try to reproduce this and then see if something pops up.

I also tried to reproduced it, but without success.

To avoid any problems with escaping characters, quotes, etc.: Should we introduced an output option to the ls command which will print the query at first and then prints the results (and a summary maybe)

```
beet ls path:"/path/to/library/A/File.mp3"
```

could give an output like

```
Searching for:
path:"/path/to/library/A/File.mp3"

Results:
Beatles - Let it be - Let it be

Summary:
Found 1 track
```

What do you think?

Some additional debugging output might serve this role well too.

I too think it could be a useful debug info to have but that it indeed wouldn't belong to every day output.

After digging a bit more into the code I found something which might help to understand what's going on here. I guess the root of the problem is the type of the `path`-column at the database: It's a BLOB. Using a BLOB column within a where clause can be tricky sometimes.
I've changed the where clause from

```
({0} = ?) || ({0} LIKE ? ESCAPE ?)
```

to

```
(lower({0}) = lower(?)) || (lower({0}) LIKE lower(?) ESCAPE ?)
```

and now it seems to work for me. But I'm really unsure if this is the right solution.

Any comments?

Aha! I think so! In the case-insensitive case (sorry), the pattern is lower-cased: https://github.com/sampsyo/beets/blob/master/beets/library.py#L71

So checking equality as in `{0} = ?` isn't right; we also need to lower-case the value from the database.

Does just changing it to this work?

```
(lower({0}) = ?) || ({0} LIKE ? ESCAPE ?)
```

That is, the pattern is already lower-cased, so no need for `lower(?)`. And LIKE is already case-insensitive, so leave that alone. I think we just need `lower` on the value in the equality.

No, that doesn't do the trick. But you are right with the `LIKE` part of the where clause: we don't need the `lower` there. So the following works perfect for me (at least with the test cases I've got here...):

```
(lower({0}) = lower(?)) || ({0} LIKE ? ESCAPE ?)
```

But I'm still not sure why we have to do a `lower` for the parameter since the string is already in lower case. Maybe Python and SQLite create something different when using `lower`?

Good thinking. Do you think you might be able to use print-debugging to track down a case where Python's `path.lower()` doesn't match SQLite's `lower(path)`? Then we can really nail this bug.

In any case, if there really is a difference, then perhaps we should keep `LOWER({0}) = LOWER(?)` in that query and then _remove_ the lower-casing on the Python side.

I did as you suggested: Selected `lower(path)` and used `path.lower()` in Python and printed the byte representation of each character. They are both absolutely identical.

Removing the lower-casing on the Python side seems to have no effect, hence everything is still working perfectly. I'll prepare a PR containing these changes.


Resolving patch:
diff --git a/beets/library.py b/beets/library.py
--- a/beets/library.py
+++ b/beets/library.py
@@ -92,9 +92,8 @@ def match(self, item):
         return (path == self.file_path) or path.startswith(self.dir_path)
 
     def col_clause(self):
-        file_blob = buffer(self.file_path)
-
         if self.case_sensitive:
+            file_blob = buffer(self.file_path)
             dir_blob = buffer(self.dir_path)
             return '({0} = ?) || (substr({0}, 1, ?) = ?)'.format(self.field), \
                    (file_blob, len(dir_blob), dir_blob)
@@ -102,8 +101,11 @@ def col_clause(self):
         escape = lambda m: self.escape_char + m.group(0)
         dir_pattern = self.escape_re.sub(escape, self.dir_path)
         dir_blob = buffer(dir_pattern + b'%')
-        return '({0} = ?) || ({0} LIKE ? ESCAPE ?)'.format(self.field), \
-               (file_blob, dir_blob, self.escape_char)
+        file_pattern = self.escape_re.sub(escape, self.file_path)
+        file_blob = buffer(file_pattern)
+        return '({0} LIKE ? ESCAPE ?) || ({0} LIKE ? ESCAPE ?)'.format(
+            self.field), (file_blob, self.escape_char, dir_blob,
+                          self.escape_char)
 
 
 # Library-specific field types.


Test patch: