JoKnopp/dmoz2db — pull request #7 by siccovansas (2014-08-04 14:32:04+00:00)
Title: Fixed problem with too long indices

Fixed a problem which crashed the script crashed with error: "Specified key was too long; max key length is 767 bytes"

InnoDB has a maximum index length of 767 bytes. Each unicode character takes up 3 bytes so columns with too many bytes (i.e., altlangs.resource, categories.Topic and externalpages.link) could not have their full values indexed. Instead these columns now use 'prefix indexes' which specify a maximum index length. I chose lengths which cover approximately 90-95% of the strings in the columns. The remaining strings share their index with 1 or more other strings, which causes longer look-up times, but this is probably justified as the index length would need to drastically increase to cover this minority.