Updated Sphinx API.

Updated with the file that comes with Sphinx 2.0.4.
Updated Sphinx beheaviour.

Code from Clément Hallet [1]. It has more features.

[1] https://github.com/PathMotion/cakephp-sphinx-behavior/ rev. a7a5606
Make rankingMode support multiple parameters.
Add user id and creation date in Sphinx indexes.

This way, we will be able to perform ranking and sorting based on these values
while performing searches.
Boost ranking of owned sentences.

This changes the default ranking formula for searches to make owned sentences
appear first. The results are basically divided into two parts. First all the
owned sentences and then the unowned sentences. Each part on its own is ranked
exactly like before.
Lower the ranking of unapproved sentences.

Sentences with correctness == -1 will appear at the very end of
the results, regardless of their adoption state. So the criteria
are now (in order of weight): correctness > adopted > relevance.
Added comments around values of charset_table.

So that we can get a clue of what’s going on.
Added support for Ethiopic alphabet.

In order to make search for the Amharic language possible.
Stop ignoring diacritics, better case folding.

The previous Sphinx config was discarding all the diacritics, mapping every
character with a diacritic to its non-diacritic version. However, I think this
cause harm for searching into languages in which diacritics are as important
as if it were a different letter. For instance, in French “congrès” and
“congres” are totally different words. When one is way more frequent that the
other, sentences that include the unfrequent word gets very hard to find.

Here is a real-life example on Tatoeba: there is currently a single sentence
with the word “prisé”, and 101 sentences with the word “prise”, making that
prisé sentence very very hard to find because one has to go through all the
results to sort it out. That’s precisely because diacritics are ignored. With
this commit, looking up for “=prisé” only returns the sentence that include
that exact word. Looking up for “prisé” or “prise” (without the equal sign)
still returns sentences with close words like pris/prise/prisé…

Moreover, the case folding were broken for capital letters with diactrics and
this commit also solves that problem (#389).

I also included case folding rules for “Latin extended additional” characters
by following Unicode case character properties. I don’t know the languages
that use these characters, but again, searching with diacritics and cases
should work better for them.
Add support for more languages in Sphinx #245.

Added charset tables for Cherokee, Lao, Mon, Sinhala, Tamil, Telugu
and Tibetan. The Tibetan is temporary because I’m not sure about
including marks and other various symbols.
Fixed random sentence ids.

Reverted part of 1158508bb9fe535a2045aa5230cc62370a81cc56 since
it breaks the random sentence feature.
Tibetan and Lao don’t use word boundaries.

According to Wikipedia [1], these languages don’t use word boundaries.
As documented by Sphinx [2], one must set ngrams_len to 1 for such
languages. I tested searching in Lao with the word for cat “ແມວ”.
For Tibetan, I need to make sure the charset_table is good.

[1] https://en.wikipedia.org/wiki/Category:Writing_systems_without_word_boundaries
[2] http://sphinxsearch.com/docs/current.html#conf-ngram-len
Fixed searching in Khmer.

Included a proper charset table, and configured it
as a language without word boundaries.
Fixed searching in Thai.

Included a proper charset table, and configured it
as a language without word boundaries.
Fixed searching in Punjaby using Gurmukhi script.

Added charset table for the Gurmukhi script.
Fixed search for the Gujarati language.

Added charset table for the Gujarati script.
Exclude reserved code points from Ethiopic.
Removed duplicate Thai charset entry.
Merge pull request #433 from Tatoeba/better-search

More relevant search results