Package: bifixer
bifixerTool to fix bitexts and tag near-duplicates for removal.What can BIFIXER do to your parallel corporaFixes several text issues:Fixes mojibakeTurns HTML entities into the character they representReplaces characters from wrong alphabets with the correct onesDeactivate this feature with--ignore_charactersNormalizes punctuation and spaces (deactivate this feature with--ignore_normalization)Removes HTML tags (deactivate this feature with--ignore_html)Fixes common orthographic errors for some languages:Currently: Danish, German, English, Spanish, Dutch, Norwegian, Portuguese and TurkishDeactivate this feature with--ignore_orthographyRemoves sentences with empty source or target (deactivate this feature with--ignore_empty)Fixes common tokenization issues (deactivate this feature with--ignore_detokenization)Obtains hahes of parallel sentences, in order to ease the later removal of duplicates (deactivate this feature with--ignore_duplicates)Want stronger deduplication? Make this feature to find near-duplicated sentences (ignoring casing, accents, diacritics and digits) by using the--aggressive_dedupflagLearn more in the "Tagging duplicated and near-duplicated sentences" section below.Provides better segmentation of long sentences:Choose betweenNLTKorLoomchild(SRX-rules based) segmenter modules with--segmenter(default is NLTK)Choose the minimum length (in words) you want to start segmenting at (default is 15) with--words_before_segmenting. Set it to 1 to try to segment all sentences.Deactivate this feature with--ignore_segmentationGot MONOLINGUAL text? Usemonofixer.pyinstead.CitationIf you find Bifixer useful, please consider citing the following paper:Gema Ramírez-Sánchez, Jaume Zaragoza-Bernabeu, Marta Bañón and Sergio Ortiz Rojas"Bifixer and Bicleaner: two open-source tools to clean your parallel data.",inProceedings of the 22nd Annual Conference of the European Association for Machine Translation.Lisboa, Portugal: European Association for Machine Translation, November 2020@InProceedings{prompsit:2020:EAMT,
  author    ={Gema Ram\'{i}rez-S\'{a}nchez and Jaume Zaragoza-Bernabeu and Marta Ba{\~n}\'{o}n and Sergio Ortiz-Rojas},
  title     ={Bifixer and Bicleaner: two open-source tools to clean your parallel data.},
  booktitle ={Proceedings of the 22nd Annual Conference of the European Association for Machine Translation},
  pages	    ={291--298},
  isbn      ={978-989-33-0589-8},
  year	    ={2020},
  month     ={November},
  address   ={Lisboa, Portugal},
  publisher ={European Association for Machine Translation}}INSTALLATIONInstall from source:gitclonehttps://github.com/bitextor/bifixercdbifixer
pipinstall.Automatic testing was added to ensure that everything is working fine in Bifixer:cdbifixer
pytestOr install without manually downloading the repo:pipinstall"bifixer @ git+https://github.com/bitextor/bifixer.git"Or even easier, install directly from PyPI:pipinstallbifixerAlso, you can install the conda package:condainstall-cbitextorbifixerAfter installing, two executables (bifixerandmonofixer) will be available to be run.Loomchild segmenterPlease note that, in order to use the optionalloomchildsegmenter module in Java, it has to be specified as an optional dependency during installation:pipinstallbifixer[loomchild]In case you are not using Java 8 as default, download it and overwrite the 'JAVA_HOME' variable before installing, for example:exportJAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64/USAGEBifixerusage:bifixer.py[-h][--header][--scolSCOL][--tcolTCOL][--sdeferredcolSDEFERREDCOL][--tdeferredcolTDEFERREDCOL][--ignore_characters][--ignore_empty][--ignore_long][--ignore_orthography][--ignore_detokenization][--ignore_duplicates][--aggressive_dedup][--ignore_segmentation][--ignore_html][--words_before_segmentingWORDS_BEFORE_SEGMENTING][--segmenter{nltk,loomchild}][--annotated_output][--tmp_dirTMP_DIR][-q][--debug][--logfileLOGFILE][-v]inputoutputsrclangtrglang

positionalarguments:inputTab-separatedfilestobebifixedoutputFixedcorpussrclangSourcelanguage(SL)oftheinputtrglangTargetlanguage(TL)oftheinput

optionalarguments:-h,--helpshowthishelpmessageandexitOptional:--headerInputfilewillhaveheader(default:False)--scolSCOLSourcesentencecolumn(startingin1).Thenameofthefieldisexpectedinsteadofthepositionif--headerisset(default:3)--tcolTCOLTargetsentencecolumn(startingin1).Thenameofthefieldisexpectedinsteadofthepositionif--headerisset(default:4)--sdeferredcolSDEFERREDCOLSourcedeferredstandoffannotationcolumn(startingin1).Thenameofthefieldisexpectedinsteadofthepositionif--headerisset(default:None)--tdeferredcolTDEFERREDCOLTargetdeferredstandoffannotationcolumn(startingin1).Thenameofthefieldisexpectedinsteadofthepositionif--headerisset(default:None)--ignore_charactersDoesn't fix mojibake, orthography, or other characterissues (default: False)--ignore_empty        Doesn'tremovesentenceswithemptysourceortarget(default:False)--ignore_longDoesn't ignore too long sentences (default: False)--ignore_orthography  Doesn'tapplyorthographyfixing(default:False)--ignore_htmlDoesn't remove HTML tags (default: False)--ignore_detokenizationDoesn'tfixcommontokenizationissues(default:False)--ignore_duplicatesDoesn't obtain the hashes of parallel sentences(default: False)--aggressive_dedup    Treats similar sentences as duplicates (marking themwith the same hash) (default: False)--ignore_segmentationDoesn'tchangesegmentationoflongsentences(default:False)--words_before_segmentingWORDS_BEFORE_SEGMENTINGMaxwordsallowedinonesideofaparallelsentencebeforetryingtosegmentateit.Setto0toapplicatesegmentationoneverything.(default:15)--segmenter{nltk,loomchild}Segmentermodule.(default:nltk)--annotated_outputAddsanextracolumnindicatingifthesentencepairwasmodified('Yes'ifitwasmodified,otherwise'No')(default:False)--tmp_dirTMP_DIRTemporarydirectorywherecreatingthetemporaryfilesofthisprogram(default:/tmp)Logging:-q,--quietSilentloggingmode(default:False)--debugDebugloggingmode(default:False)--logfileLOGFILEStorelogtoafile(default:<_io.TextIOWrappername='<stderr>'mode='w'encoding='UTF-8'>)-v,--versionshowversionofthisscriptandexitParametersPositional:INPUT : Input file. Tab-separated bilingual input file. By default, the expected columns are: SRC_URL TRG_URL SRC_SENTENCE TRG_SENTENCE [EXTRA COLUMNS]. When INPUT is -, reads standard input.OUTPUT : Output file. Tab-separated bilingual output file, being a fixed version of the input file. By default, the output columns are: SRC_URL TRG_URL SRC_SENTENCE TRG_SENTENCE [EXTRA COLUMNS] HASH RANKING. When OUTPUT is -, writes standard input.SRC LANG : Source language code (2-letter ISO 639-1 code)TRG LANG : Target language code (2-letter ISO 639-1 code)Optional:--tmp_dir TMP_DIR : Directory for temporary files--header : Treats the first sentence of the input file as the header row. If set, the output will contain a header as well--scol SCOL : Position of the source sentence column (starting in 1). If--headeris set, the expected value will be the name of the field. Default: 3 if--headeris not set else src_text--tcol TCOL : Position of the target sentence column (starting in 1). If--headeris set, the expected value will be the name of the field. Default: 4 if--headeris not set else trg_text--sdeferredcol SDEFERREDCOL : Source deferred standoff annotation column (starting in 1). Default: None--tdeferredcol TDEFERREDCOL : Target deferred standoff annotation column (starting in 1). Default: None--ignore_duplicates : Deactivates deduplication (won't add hash or ranking)--ignore_empty : Doesn't remove sentences with empty source or target--ignore_long: Doesn't remove too long sentences--ignore_html: Doesn't remove HTML tags--ignore_segmentation : Deactivates segmentation of long sentences--segmenter: Segmenter module (nltkorloomchild). Default: nltk--words_before_segmenting : Maximum allowed amount of words in a sentence, before trying to segment it. Default: 15--ignore_characters : Deactivates text fixing (characters, encoding...)--ignore_orthography  Deactivates orthography fixing--ignore_detokenization : Doesn't fix common tokenization issues.--aggressive_dedup : Treats near-duplicated sentences as duplicates (normalizes sentences before hashing)--annotated_output    Adds an extra column indicating if the sentence pair was modified ('Yes' if it was modified, otherwise 'No'). Default: False--tmp_dir TMP_DIR : Directory for temporary files-q, --quiet : Silent logging mode--debug: Shows debug messages while running--logfile LOGFILE : Stores log into a file-v, --version : Shows version number and exits-h, --help: Shows help and exitsMonofixerpython3.7bifixer/monofixer.py--help
usage:monofixer.py[-h][--scolSCOL][--sdeferredcolSDEFERREDCOL][--ignore_characters][--ignore_long][--ignore_orthography][--ignore_detokenization][--ignore_duplicates][--aggressive_dedup][--ignore_segmentation][--ignore_html][--words_before_segmentingWORDS_BEFORE_SEGMENTING][--segmenter{nltk,loomchild}][--annotated_output][--tmp_dirTMP_DIR][-q][--debug][--logfileLOGFILE][-v]inputoutputlang

positionalarguments:inputTab-separatedfiletobefixedoutputFixedcorpuslangLanguageoftheinput

optionalarguments:-h,--helpshowthishelpmessageandexitOptional:--headerInputfilewillhaveheader(default:False)--scolSCOLSentencecolumn(startingin1).Thenameofthefieldisexpectedinsteadofthepositionif--headerisset(default:2)--sdeferredcolSDEFERREDCOLSourcedeferredstandoffannotationcolumn(startingin1).Thenameofthefieldisexpectedinsteadofthepositionif--headerisset(default:None)--ignore_charactersDoesn't fix mojibake, orthography, or other characterissues (default: False)--ignore_long         Doesn'tignoretoolongsentences(default:False)--ignore_orthographyDoesn't apply orthography fixing (default: False)--ignore_detokenizationDoesn'tfixcommontokenizationissues(default:False)--ignore_htmlDoesn't remove HTML tags (default: False)--ignore_duplicates   Doesn'tobtainthehashesofsentences(default:False)--aggressive_dedupTreatssimilarsentencesasduplicates(markingthemwiththesamehash)(default:False)--ignore_segmentationDoesn't change segmentation of long sentences(default: False)--words_before_segmenting WORDS_BEFORE_SEGMENTINGMax words allowed in a parallel sentence before tryingto segmentate it. Set to 0 to applicate segmentationon everyt33hing. (default: 15)--segmenter {nltk,loomchild}Segmenter module. (default: nltk)--annotated_output    Adds an extra column indicating if the sentence  wasmodified ('Yes' if it was modified, otherwise 'No')(default: False)--tmp_dir TMP_DIR     Temporary directory where creating the temporary filesof this program (default: /tmp)Logging:-q, --quiet           Silent logging mode (default: False)--debug               Debug logging mode (default: False)--logfile LOGFILE     Store log to a file (default: <_io.TextIOWrappername='<stderr>' mode='w' encoding='UTF-8'>)-v,--versionshowversionofthisscriptandexitParametersPositional:INPUT : Input file. Tab-separated monolingual input file. By default, the expected columns are: URL SENTENCE [EXTRA COLUMNS]. When INPUT is -, reads standard input.OUTPUT : Output file. Tab-separated monolingual output file, being a fixed version of the input file. By default, the output columns are: URL SENTENCE [EXTRA COLUMNS] HASH RANKING. When OUTPUT is -, writes standard input.LANG : Sentence language code (2-letter ISO 639-1 code)Optional:--tmp_dir TMP_DIR : Directory for temporary files--header : Treats the first sentence of the input file as the header row. If set, the output will contain a header as well--scol SCOL : Position of the source sentence column (starting in 1). If--headeris set, the expected value will be the name of the field. Default: 2 if--headeris not set else src_text--sdeferredcol SDEFERREDCOL  Sentence deferred standoff annotation column (starting in 1). Default: None--ignore_duplicates : Deactivates deduplication (won't add hash or ranking)--ignore_long: Doesn't remove too long sentences--ignore_html: Doesn't remove HTML tags--ignore_segmentation : Deactivates segmentation of long sentences--segmenter: Segmenter module (nltkorloomchild). Default: nltk--words_before_segmenting : Maximum allowed amount of words in a sentence, before trying to segment it. Default: 15--ignore_characters : Deactivates text fixing (characters, encoding...)--ignore_orthography : Deactivates orthography fixing--ignore_detokenization : Doesn't fix common tokenization issues.--aggressive_dedup : Treats near-duplicated sentences as duplicates (normalizes sentences before hashing)--annotated_output    Adds an extra column indicating if the sentence was modified ('Yes' if it was modified, otherwise 'No'). Default: False--tmp_dir TMP_DIR : Directory for temporary files-q, --quiet : Silent logging mode--debug: Shows debug messages while running--logfile LOGFILE : Stores log into a file-v, --version : Shows version number and exits-h, --help: Shows help and exitsRUNSingle thread runbifixerinput-corpus.en-esoutput-corpus.en-esenesRunning in parallelbifixercan be parallelized by using your favourite method (for example, GNU parallel)Suggested usage:catinput-corpus.en-es\|parallel-j25--pipe-k-l30000bifixer-q--enes\>output-corpus.en-eswhere the two '-' mean read from stdin and write to stdout, and the-qtells bifixer to be quiet in order to avoid logging a lot of information messages.TAGGING DUPLICATED AND NEAR-DUPLICATED SENTENCESIn order to ease the later removal of duplicated or near-duplicated parallel sentences, Bifixer appends each parallel sentence two new fields:hashandranking.The hash is obtained by using theXXHashalgorithm, applied after fixing source and target sentences  (fixed_source+"\t"+fixed_target). Sentences that are identical at this step (see example below) will get the same hash.When using the--aggressive_dedupfeature, fixed parallel sentences are also normalized (ignoring casing, accents and diacritics) before their hash is computed. Doing so, sentences that are near-duplicates (i.e. they only differ in casing or accents) will also get the same hash. Normalization is only used internally: the output sentences will not be normalized after Bifixer is applied.Arankingcolumn is added at the end of each line. When not using the--aggressive_dedupfeature, the number is set to 1 by default. When using the--aggressive_dedupfeature, a float number is provided. This number (interpreted as the higher the better) will be used at later step to help the deduplication algorithm to choose the best sentence from those sharing the same hash. If the ranking number is exactly the same for a group of sentences sharing the same hash, only a random one should be kept. Otherwise, the one with the highest ranking number should be kept.EXAMPLEInput file:http://www.ehyz.com/2.html.tmp	http://www.ehyz.com/2.html.tmp	1 year ago NuVid	Hace 1 aÃ±o NuVid
http://pandafoundation.com/index.php?page=7	http://pandafoundation.com/index.php?page=26	©2007 Chengdu Research Base of Giant Panda Breeding ! All Rights Reserved	©2017 Fundación para la Investigación de Cría del Panda Gigante de Chengdu/ ¡Todos los derechos reservados!     
http://www.boliviamall.com/4520.html	http://www.boliviamall.com/4520.html	Welcome Guest 1! Would you like to log in ?	Bienvenido Invitado 1! ¿Le gustaria entrar ?    
http://pandafoundation.com/index.php?page=157	http://pandafoundation.com/index.php?page=76	©2007 Chengdu Research Base of Giant Panda Breeding ! All Rights Reserved	©2017 Fundación para la Investigación de Cría del Panda Gigante de Chengdu/ ¡Todos los derechos reservados!
http://www.ehyz.com/6.html.tmp	http://www.ehyz.com/6.html.tmp	1 year ago NuVid	Hace 1 año NuVid
http://www.boliviamall.com/4305.html	http://www.boliviamall.com/4305.html	Welcome Guest 12! Would you like to log in?	¡Bienvenido invitado 12! ¿Le gustaria entrar?Output file (using the '--aggressive_dedup' feature, otherwise ranking number would be 1 in all cases):http://www.ehyz.com/2.html.tmp	http://www.ehyz.com/2.html.tmp	1 year ago NuVid	Hace 1 año NuVid	9f1f7c6fc775a23a	88.25
http://pandafoundation.com/index.php?page=7	http://pandafoundation.com/index.php?page=26	©2007 Chengdu Research Base of Giant Panda Breeding ! All Rights Reserved	©2017 Fundación para la Investigación de Cría del Panda Gigante de Chengdu/ ¡Todos los derechos reservados!	d0278d1279f06823	91.93
http://www.boliviamall.com/4520.html	http://www.boliviamall.com/4520.html	Welcome Guest 1! Would you like to log in ?	Bienvenido Invitado 1! ¿Le gustaría entrar ?	e8f129b1624b9f5d	91.22
http://pandafoundation.com/index.php?page=157	http://pandafoundation.com/index.php?page=76	©2007 Chengdu Research Base of Giant Panda Breeding ! All Rights Reserved	©2017 Fundación para la Investigación de Cría del Panda Gigante de Chengdu/ ¡Todos los derechos reservados!	d0278d1279f06823	91.93
http://www.ehyz.com/6.html.tmp	http://www.ehyz.com/6.html.tmp	1 year ago NuVid	Hace 1 año NuVid	9f1f7c6fc775a23a	88.25
http://www.boliviamall.com/4305.html	http://www.boliviamall.com/4305.html	Welcome Guest 12! Would you like to log in?	¡Bienvenido invitado 12! ¿Le gustaría entrar?	422aeefd8f056b30	92.78All documents and software contained in this repository reflect only the authors' view. The Innovation and Networks Executive Agency of the European Union is not responsible for any use that may be made of the information it contains.