Jarn/collective.solr — Issue #10: Editing search_pattern through the control panel results in potential UnicodeDecodeErrors on search
Author: mjpieters
Created: 2012-01-27 15:01:05+00:00
State: open

The search_pattern parameter is used to turn a Plone search form query into a SOLR query; the (byte string, encoded to utf8) search is interpolated into this parameter to be sent to SOLR.

This goes great if the search_pattern parameter is set through portal_setup; the parameter is then set as a python string (so encoded, not a unicode string). However, when you edit the parameter through the control panel form, the value is stored as a unicode string.

Once this has happened, all non-ascii searches will fail with a UnicodeDecodeError, as Python will try and decode the utf-8 search byte string with ASCII to form a unicode string to interpolate into the unicode search_pattern parameter.

The error takes place in collective.solr.mangler:

```python
# ... start of TB elided ...
Module collective.indexing.monkey, line 84, in searchResults
Module collective.solr.monkey, line 31, in searchResults
Module collective.solr.dispatcher, line 42, in __call__
Module collective.solr.dispatcher, line 86, in solrSearchResults
Module collective.solr.mangler, line 90, in mangleQuery
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)
```
We need to either that both query and parameter are byte strings or unicode, not one or the other.