twain47/Nominatim — pull request #28 by lonvia (2012-09-26 06:41:14+00:00)
Title: ignore frequent partial search terms

This removes frequent partial search terms from the search_name table and the list of words handled during search.

Adds word counts from a full planet to the word table. It would be possible to generate the word counts from the imported database but loading pregenerated counts is slightly faster and this way extracts work with the same numbers as the full planet.

There is a new configuration option CONST_Max_Word_Frequency which allows to
take into account the word count. This works differently for import/update and search.

For import/updates, only the value set during initial imports is taken into account. It determines if a word is added to search_name. (via a modified getorcreate_word_id function).

For search requests, the value during runtime determines if a single term should be
used or simply be ignored. This is pretty much what has been done on poldi already. The difference was that the blacklisting of words was done manually.

A test import of the planet is still in progress but I thought I leave the code here for comments already.