owncloud/core — pull request #849 by butonic (2012-12-12 21:45:51+00:00)
Title: Search lucene simple

This is more of a proof of concept implementation than a serious pull request. Nevertheless, I'd like to get some feedback on this as soon as possible.

## Getting it to work
First switch your 3rdparty repo to the [search_lucene_simple_required_zend_libs](https://github.com/owncloud/3rdparty/tree/search_lucene_simple_required_zend_libs) branch for the required Zend libraries.
Then, in this search_lucene_simple branch you can enable the "Lucene Search" App that will allow you to do full text search in text, HTML and PDF (see issue #162) files.

# Beware: it completely removes any other search providers in this testing phase.
You will not be able to search for contacts or calendar entries while the app is enabled.

New files beeing uploaded will automagically be indexed. For a full rescan got to the settings view and do a reindex.

## Searching
There is a minor regression regarding the queries. By default the core search uses a wildcard before and after the query term to match partial strings. Lucene can work like this but it will be painfully slow. On the upside you can do negations and all kinds of [query magic](http://lucene.apache.org/core/old_versioned_docs/versions/2_9_1/queryparsersyntax.html) Searching for a partial term like 'term*' is ok performance wise. '*term*' ... well ... it works ... but it takes ages and you should know what you are doing.

## Testing
I am testing this with the [zipped package of linux howtos](http://www.tldp.org/pub/Linux/docs/HOWTO/Linux-html-HOWTOs.tar.gz) from [The Linux Documentation Project](http://www.tldp.org/docs.html#howto). aside from the html files it also has the PDF versions and contains more than 8000 html files. Be warned: indexing all these files (143MB) took about 5 hours on my machine and produced an index of <25MB.

## Troubleshooting
In case you think something is wrong you can just start over by deleting the users lucene index folder: `.../path/to/data/user/lucene_index` and pushing the rescan button on the personal settings page.

## Pros & Cons
The implementation uses Zend_Search_Lucene which has pros and cons:
+ plain php
+ powerful query language
+ based on the [lucene](http://en.wikipedia.org/wiki/Lucene) standard
+ search index can be examined with [luke](http://code.google.com/p/luke/)
- slow
- unbearably slow when using wildcards in front of terms like '*term' so cannot exactly replace the current search

## Sidenotes

Along with this I rewrote the file cache scan in the spirit of a visitor pattern with a walk method (which I should have submitted as a separate pull request, yes). A replacement implementation for the old scan method exists and a new indexer visitor is implemented to reindex all files from the personal settings page.

I already discussed "search as filter" with @jancborchardt (also see #821) but there is still a long way to go. For the short term I think I need something to merge search results in the browser so that I can skip adding a file that has already been found by the core file search provider and get rid of the current "throw out all the other search providers".

I'd really like to add an abstraction over the indexing implementation so that the ownCloud search API not only allows adding search providers but also changing the back end implementation. This would allow apps to say "hey index these meta fields" and not have to implement the search themselves.