IT WORKS! Construction and searching by a table of ngrams, comparison by string distance, search results.
Heuristics still very much in need of tweaking.
Sane weighting for ngram matches.
 - Changed search result weighting to float
 - Weighting as segment of mx+c weighting
 - Store correct length melody (was fixed-size, not always accurate)
Shaved 120 MB off RAM usage (from 3.1 GB) by storing melody as int8
Shrunk some types in search result type.
Mitigate effect of tune length on tune weighting.
Constrain tune search by only looking in areas where ngrams were found. Up to 2x speedup.
Decouple the ngram identification process from the filtering process.
Speedup and memory saving. Memory down from 3.00 GB to 1.85 GB.
 - Copy and trim melody so the slice isn't hanging onto a pointer to an over-large array
 - Change types to smaller ones
   - startEndInterval fields from uint to uint16
   - docId from int to uint32
Parallel results
 - Introduce goroutines to the ngram string distance filtering process.

Still quite slow, around 7 seconds to query.
Got melody search based on Ngram and string distance working functionally.

 - Slimmed search to two steps, ngram identification and then similarity rating.
   Removed intermediate take-top similarity rating and related stuff.
 - Stripping of ngram occurrence intervals that are no-hopers.
 - Basic weighting on effective matched length multiplied by primacy within tune.
 - Consider each ngram occurrence interval on its own (i.e. many occurrences per tune) then pick best
   per tune rather than glom all occurrences together (recipe for combintorial explosion in the similarity).
 - Remove triangular area weighting equation, this one seems good enough.

 - Added histogram similarity, but not using it in the end. It'll come in useful later.

 - Remove goroutines and channels, it only cluttered and didn't make a massive improvement.
 - Tidyup of dead code
Trimmed some recursion in string distance matching, about a 2x speedup (between 13x and 1x).
Added gaps to ngrams for approximate matching.
Increased size of ngram
Fixed fuzzy matching in the string distance function.
Housekeeping and memory slimmage.
Slimmed startEndIntervals down from 48 bytes to 16.
Scrapped unused methods for startEndIntervals types
Scrapped 'count' in startEndIntervals
Added melodyOffset type to replace various uints.
Added explicit type checking for type conversions to melodyOffset
Added table size to info API response.
Removal of duplicates based on interval similarity and then string distance.
Reduced size to about half. 100618 duplicates out of 194886 tunes.
Removal of duplicates based on interval similarity and then string distance.
Reduced size to about half. 100618 duplicates out of 194886 tunes.
Merge branch 'ngram' of github.com:afandian/tunesearch into ngram

Conflicts:
	corpus.go
	filesystem.go
Introduce histogram similarity measure as interrim ranking measure betwen ngram and string distance.
Weighting and hashing improvements.
Changed comparison of ResultSet to use weight not length (as length is accounted for in weight).
Turned ngram hashcode into a type ngram_t.
After experimentation with 8 and 16 bit hashes, stuck with 32 bit and improved hash function.
Found happy medium.
 - 2.4 GB RAM, most requests under 200ms.
 - ngram length 6
 - filter histogram distance at 1
 - position-skewed histogram
 - ngram 32 bits, non-lossy hash
 - stack-allocated histogram
Change storage of ngram occurrences from interval pairs to single offets.
Intervals insertion tweaks to keep RAM down.
Reduce size of hash to 16 bits, reduced ram to acceptable 2GB.