Repository: carpentries/amy

Issue:
search fails because we search fields separately
A search for 'Greg Wilson' turns up nothing, because 'Greg Wilson' isn't found in forenames, surnames, or site names.  A search for 'Wilson' finds Greg Wilson, plus other Wilsons, because 'Wilson' is found in a surname field.



Discussion:
On Wed, May 27, 2015 at 07:12:32PM -0700, Greg Wilson wrote:

> A search for 'Greg Wilson' turns up nothing, because 'Greg Wilson'
> isn't found in forenames, surnames, or site names.  A search for
> 'Wilson' finds Greg Wilson, plus other Wilsons, because 'Wilson' is
> found in a surname field.

Another reason for #66 ;).  But baring that, I guess we need a
tokenizer to split search terms (maybe shlex.split? [1](https://docs.python.org/3/library/shlex.html#shlex.split)) and then
build a set of or'ed Q objects?  Fun, fun :p.  Alternatively have
separate search widgets for each searchable field, so folks can put
“Wilson” in the surname-search box and “Greg” in the forename search
box.

Yeah, eventually we have to do real search.  For now, understanding that 
it works field by field will let admins do what they need to, and we 
don't have so much data that it's impossible to find things (yet).

I think we could try ouy the [Haystack](http://haystacksearch.org/) + [Whoosh](https://bitbucket.org/mchaput/whoosh/wiki/Home) combo. @wking what do you think about that?

I used H+W for some complex task on a big database once and it was really slow compared to Solr, but for our needs it could work, I think.

My past experience with Haystack + Whoosh wasn't inspiring - as you 
said, it's slow.  Let's defer this until 0.5 at the earliest.
Thanks,
Greg

On 2015-06-05 12:15 PM, Piotr Banaszkiewicz wrote:

> I think we could try ouy the Haystack http://haystacksearch.org/ + 
> Whoosh https://bitbucket.org/mchaput/whoosh/wiki/Home combo. @wking 
> https://github.com/wking what do you think about that?
> 
> I used H+W for some complex task on a big database once and it was 
> really slow compared to Solr, but for our needs it could work, I think.
> 
> —
> Reply to this email directly or view it on GitHub 
> https://github.com/swcarpentry/amy/issues/319#issuecomment-109347852.

## 

Dr. Greg Wilson    | gvwilson@software-carpentry.org
Software Carpentry | http://software-carpentry.org

On Fri, Jun 05, 2015 at 09:15:25AM -0700, Piotr Banaszkiewicz wrote:

> I think we could try ouy the Haystack + Whoosh combo. @wking what do
> you think about that?

I haven't used Haystack + Whoosh myself, but for work I'm probably
going to be working on moving out full-text search into our main
PostgreSQL database in the next few weeks [1,2].  I'm not sure which
database we're using in production, or whether we need to keep SQLite
support for local testing (I'm fine just not testing the search
support if we don't have a local PostgreSQL database), but if
PostgreSQL is an option that will be a full-text search approach that
I'll be familiar with soon ;).

@wking:

>  I'm not sure which database we're using in production

We're using SQLite in production (both Small Orange and Rackspace).

> whether we need to keep SQLite support for local testing

For now I'd say yes for SQLite locally. We could maybe use PostgreSQL for development if Vagrant / Docker / Hyper solution would be very easy to set up on devs machines (which consist of MacOS X and Linux as far as I know).

I'll try to not forget about testing haystack + whoosh locally. Maybe they can perform well for our relatively small needs.

Update:

I worked on a auxiliary function that builds a query from tokenized terms:

``` python
terms = re.split('\s+', 'piotr banaszkiewicz')
def build_OR_query(model, terms, *fields):
    query = Q()
    for term in terms:
        for field in fields:
            arg = {
                field + '__contains': term
            }
            query |= Q(**arg)
    return model.objects.filter(query)
```

But the results are kind-of disappointing:
1. Searching for 'greg wilson' will yield everyone named Greg or Wilson.
2. We have no result rating (like real full-text search does) so it's hard to pick a result with good value for the user

The bottom line is: with this code searching for "Greg Wilson" it's no different from searching for "Greg" (or "Wilson").

Bummer :-(


Resolving patch:
diff --git a/workshops/views.py b/workshops/views.py
--- a/workshops/views.py
+++ b/workshops/views.py
@@ -969,12 +969,17 @@ def search(request):
         form = SearchForm(request.POST)
         if form.is_valid():
             term = form.cleaned_data['term']
+            tokens = re.split('\s+', term)
+            results = list()
+
             if form.cleaned_data['in_sites']:
                 sites = Site.objects.filter(
                     Q(domain__contains=term) |
                     Q(fullname__contains=term) |
                     Q(notes__contains=term)) \
                     .order_by('fullname')
+                results += list(sites)
+
             if form.cleaned_data['in_events']:
                 events = Event.objects.filter(
                     Q(slug__contains=term) |
@@ -982,18 +987,38 @@ def search(request):
                     Q(site__domain__contains=term) |
                     Q(site__fullname__contains=term)) \
                     .order_by('-slug')
+                results += list(events)
+
             if form.cleaned_data['in_persons']:
-                persons = Person.objects.filter(
-                    Q(personal__contains=term) |
-                    Q(family__contains=term) |
-                    Q(email__contains=term) |
-                    Q(github__contains=term)) \
-                    .order_by('family')
+                # if user searches for two words, assume they mean a person
+                # name
+                if len(tokens) == 2:
+                    name1, name2 = tokens
+                    complex_q = (
+                        Q(personal__contains=name1) & Q(family__contains=name2)
+                    ) | (
+                        Q(personal__contains=name2) & Q(family__contains=name1)
+                    ) | Q(email__contains=term) | Q(github__contains=term)
+                    persons = Person.objects.filter(complex_q)
+                else:
+                    persons = Person.objects.filter(
+                        Q(personal__contains=term) |
+                        Q(family__contains=term) |
+                        Q(email__contains=term) |
+                        Q(github__contains=term)) \
+                        .order_by('family')
+                results += list(persons)
+
             if form.cleaned_data['in_airports']:
                 airports = Airport.objects.filter(
                     Q(iata__contains=term) |
                     Q(fullname__contains=term)) \
                     .order_by('iata')
+                results += list(airports)
+
+            # only 1 record found? Let's move to it immediately
+            if len(results) == 1:
+                return redirect(results[0].get_absolute_url())
 
     # if a GET (or any other method) we'll create a blank form
     else:


Test patch:
diff --git a/workshops/test/test_search.py b/workshops/test/test_search.py
--- a/workshops/test/test_search.py
+++ b/workshops/test/test_search.py
@@ -1,5 +1,5 @@
 from django.core.urlresolvers import reverse
-from ..models import Site
+from ..models import Site, Person
 from .base import TestBase
 
 
@@ -28,22 +28,22 @@ def test_search_for_site_when_site_matching_turned_off(self):
     def test_search_for_site_by_partial_name(self):
         response = self.client.post(reverse('search'),
                                     {'term' : 'Alpha',
-                                     'in_sites' : 'on'})
-        doc = self._check_status_code_and_parse(response, 200)
-        node = self._get_1(doc, ".//a[@class='searchresult']",
-                           'Expected exactly one search result')
-        assert node.text=='alpha.edu', \
-            'Wrong name "{0}" in search result'.format(node.text)
+                                     'in_sites' : 'on'},
+                                    follow=True)
+        assert response.status_code == 200
+        content = response.content.decode('utf-8')
+        # no way for us to check the url…
+        assert str(self.site_alpha) in content
 
     def test_search_for_site_by_full_domain(self):
         response = self.client.post(reverse('search'),
                                     {'term' : 'beta.com',
-                                     'in_sites' : 'on'})
-        doc = self._check_status_code_and_parse(response, 200)
-        node = self._get_1(doc, ".//a[@class='searchresult']",
-                           'Expected exactly one search result')
-        assert node.text=='beta.com', \
-            'Wrong name "{0}" in search result'.format(node.text)
+                                     'in_sites' : 'on'},
+                                    follow=True)
+        assert response.status_code == 200
+        content = response.content.decode('utf-8')
+        # no way for us to check the url…
+        assert str(self.site_beta) in content
 
     def test_search_for_site_with_multiple_matches(self):
         response = self.client.post(reverse('search'),
@@ -56,3 +56,23 @@ def test_search_for_site_with_multiple_matches(self):
         texts = set([n.text for n in nodes])
         assert texts == {'alpha.edu', 'beta.com'}, \
             'Wrong names {0} in search result'.format(texts)
+
+    def test_search_for_people_by_personal_family_names(self):
+        """Test if searching for two words yields people correctly."""
+        # let's add Hermione Granger to some site's notes
+        # this is required because of redirection if only 1 person matches
+        self.site_alpha.notes = 'Hermione Granger'
+        self.site_alpha.save()
+
+        response = self.client.post(reverse('search'), {
+            'term': 'Hermione Granger',
+            'in_sites': 'on',
+            'in_persons': 'on',
+        })
+        doc = self._check_status_code_and_parse(response, 200)
+        nodes = self._get_N(doc, ".//a[@class='searchresult']",
+                            'Expected two search results',
+                            expected=2)
+        texts = set([n.text for n in nodes])
+        assert texts == {str(self.site_alpha), str(self.hermione)}, \
+            'Wrong names {0} in search result'.format(texts)