Constructing Su x Arrays of Large Texts

ConstructingSuxArraysofLargeTextsK.SadakaneandH. ImaiDepartmentofInformationScience,UniversityTokyo7-3-1Hongo,Bunkyo-ku,Tokyo113-0033,JAPANfsada,imaig@is.s. u-tokyo.ac. jpAbstractRecently,Sadakane[12]prop osesanewfastandmemoryecientalgorithmforsortingsuxesofatextinlexicographicorder.Itisimp ortanttosortsuxesb ecauseanarrayofindexessuxesiscalledsuf- xarrayanditisamemoryecientalternativeofthesuxtree.SortingsuxesisalsousedforBurrows-WheelertransformationintheBlo ckSortingtextcompression,thereforefastsortingalgorithmsaredesired.Infull-textdatabases,ofcoursethelengthtextsarequitelarge,andthisalgorithmmakesitp os-sibletousethesuxarraydatastructureandcompressionschemeforsuchlargertexts.Inthispap er,wecomparealgorithmsformakingsuxarraysofBentley-Sedgewick,Andersson-NilssonandKarp-Miller-Rosenb ergmakingsuxtreesofLarssononsp eedandrequiredmemorycomparethemwithournewalgorithmwhichisfastandmem-oryecientbycombiningthem.1Intro duction1.1SuxarrayTo daylargedatabasesb ecomeavailable,suchasfulltextofnewspap ersorWebpages,andGenomesequences,thereforeitisimp ortanttostorethemonmemoryforquickqueries.Thesuxarray[10]isacompactdatastructureforon-linestringsearches.Forsuchpurp oses,suxtreeisusedb ecauseitenablesusto ndthelongestsubstringofalargetextthatmatchesthequerystringinlineartime.However,suxtreerequireshugememoryandthereforeitisimp ossibletouseitforlargetext.Thoughthesuxtreecanb econstructedinlin-eartime[11,14,9],thecomplexitydep endsonalphab etsize.Searchingtimealsodep endsontheal-phab etsize,anditb ecomesslowifthealphab etsizeislarge.Ontheotherhand,constructionandsearchingtimeofthesuxarraydo esnotdep endonsizealphab et.Thereforesearchingtimeofthesuxarraymayb ecomparablewiththesuxtreethoughithassup erlineartimecomplexityofO(Plogn)whereisthelengthofaquerystringandnislargetext.1. 2Blo ckSortingdatacompressionTheBlo ckSortingisalosslessdatacompressionscheme[4]. Thoughitscompressionratioiscompa-rablewithvariantsofthePPM[6],itscompressionsp eedisfasterthanthemanddecompressionmuchfasterthancompression.Moreover,therequiredmemoryissmallerthanthePPM,thereforeblo cksortingisapromisingscheme.TheBurrows-WheelertransformationusedintheBlo ckap ermutationoftexttob ecom-pressedanditrequiressortingofallsuxestext.ThoughtheBlo ckSortingisfasterthanPPM,itisslowerthangzipb ecausesortinglongtextisrequired.ThereforetheBlo ckSortinggenerallydi-videsalongtextinmanblocksp erformsBurrows-Wheelertransformationtoeacblo ckandthatmakesthecompressionratioorse.Thereforesp eeding-upsortingsuxesforlargetextsisim-p ortant.1. 3OurresultsWeprop oseafastandmemoryecientalgorithmforsortingallsuxesofatext.Weusethedou-blingtechniqueofKarp-Miller-Rosenb erg[8],ternarypartitioningofBentley-Sedgewick[3],andnumb er-ingmetho dofAndersson-Nilsson[1]andManb er-Myers[10]. Itrequiresonly9nbytesforsortingatextoflengthnbyusingacleverorderingsortingandanintegerenco ding.Notethatweassumethesizeofintegersis32bits. Wede neameasuredi-cultyofsortingsuxes:averagematchlength(AML). OuralgorithmrequiresmuchmorememorythantheBentley-Sedgewick,whichrequires5nbytes,butouralgorithmisfasterthanothersortingalgorithmswhenAMLislarge.Itfasterthanthesuxtreeconstruc-tionalgorithmofLarsson[9]ifthealphab etsizeislargeanditrequireslessthanhalfofmemoryforthesuxtree.Whenatextb ecomeslong,manypairsofwordsapp earrep eatedlyandtheAMLb ecomeslarge,thereforeouralgorithmispracticalforlargetexts.1. 4De nitionsAssumethatwemakasuxarrayoftextX=x[0::n1]=01: : : xn.Thesizeofalphab etisconstant.Weaddasp ecialsymb ol`$'atthetailofthetextX,thatis,xn=$. Itisnotinalphab etandsmallerthananyothersymb olsinthealphab et.SuxesofthetextarerepresentedbySi=x[i::n]. suxarrayI[0::n]anofindexessuxesSi.Allsuxesinthearrayaresortedlexicographic1