Package: easylaser
EasyLaserThis package is created to use simplyLASERfrom MetaAI to create embeddings. It uses list of string as input and returns list of numpy arrays as output instead of using files. It also does not require external tools to be installed. The package automatically downloads the required laser models.Get startedInstall with pippipinstalleasylaserBuild from sourcegitclonehttps://gitlab.com/linguacustodia/easylaser.gitcdeasylaser
pipinstall.Simple embeddings creationfromeasylaserimportLasersentences=["This is a sentence","this is another sentences."]withLaser()aslaser:embeddings=laser.embed_sentences(sentences=sentences)By default, it will try to run on the first gpu if it's available, if you don't have a gpu it will switch back to CPU.EmbeddingsSo as we have seen you can use it with a context manager or withoutfromeasylaserimportLasersentences=["This is a sentence","this is another sentences."]# with context managerwithLaser()aslaser:embeddings=laser.embed_sentences(sentences=sentences)# withoutlaser=Laser()laser.is_encoder_active()# return Falselaser.start_encoder()embeddings=laser.embed_sentences(sentences=sentences)laser.is_encoder_active()# return Truelaser.stop_encoder()Multi GPUYou can specify the hardware you want to run on :fromeasylaserimportLaserlaser=Laser(device="cpu")laser=Laser(device="cuda")laser=Laser(device=["cuda:0","cuda:1"])If you specify multiple graphic card, the inference will be multi-processed, leading to speed gain.CAUTION : There is know bug, see Issues below, with multiple graphic card, one of the parent function which use Laser should be called fromif __name__ == '__main__':From our test the relation between the number of gpu and the speed is sub-linear. If you have some ideas to improve the speed, please contact us.AlignementEmbeddings and AlignementfromdatolaserimportLaserenglish_sentences=["A cat","This is a sentence","this is another sentences."]french_sentences=["C'est une phrase","Un chat","c'est une autre phrase."]withLaser()aslaser:aligned_sentences=laser.align_sentences(english_sentences,french_sentences,threshold_score=0,keep_bad_matched=False)Every sentences, below the threshold will be considered as bad_matched.If keep_bad_matched isTrue, it keep sentence with no match as(sentence_1, None,0), if set toFalseit removes them.Only alignementYou can usealign_with_embeddingsif you have embeddings and just want to align sentencesfrom easylaser import align_with_embeddings

align_with_embeddings(
    embeddings_lang0,
    embeddings_lang1,
    sentences_lang0,
    sentences_lang1,
    threshold_score=0,
    keep_bad_matched=False,
)IssuesBecause of anissuewith faiss this package cannot go above pyhton 3.10.If you encounter the following error:RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.You might need to use thisstrutuctureto used embed_sentences with multiple GPUsdefmain()# do something hereif__name__=='__main__':main()Supported languagesLASER2The LASER2 model was trained on the following languages, so you don't need to specify a lang for these languages:Afrikaans, Albanian, Amharic, Arabic, Armenian, Aymara, Azerbaijani, Basque, Belarusian, Bengali,
Berber languages, Bosnian, Breton, Bulgarian, Burmese, Catalan, Central/Kadazan Dusun, Central Khmer,
Chavacano, Chinese, Coastal Kadazan, Cornish, Croatian, Czech, Danish, Dutch, Eastern Mari, English,
Esperanto, Estonian, Finnish, French, Galician, Georgian, German, Greek, Hausa, Hebrew, Hindi,
Hungarian, Icelandic, Ido, Indonesian, Interlingua, Interlingue, Irish, Italian, Japanese, Kabyle,
Kazakh, Korean, Kurdish, Latvian, Latin, Lingua Franca Nova, Lithuanian, Low German/Saxon,
Macedonian, Malagasy, Malay, Malayalam, Maldivian (Divehi), Marathi, Norwegian (Bokmål), Occitan,
Persian (Farsi), Polish, Portuguese, Romanian, Russian, Serbian, Sindhi, Sinhala, Slovak, Slovenian,
Somali, Spanish, Swahili, Swedish, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Turkish, Uighur,
Ukrainian, Urdu, Uzbek, Vietnamese, Wu Chinese and Yue Chinese.It has also observed that the model seems to generalize well to other (minority) languages or dialects, e.g.Asturian, Egyptian Arabic, Faroese, Kashubian, North Moluccan Malay, Nynorsk Norwegian, Piedmontese, Sorbian, Swabian, Swiss German or Western Frisian.LASER3You can also use laser on other languages in the list laser3_langs in lib/constants.py by using the lang parameter.withLaser(lang="zul_Latn")aslaser:embeddings=laser.embed_sentences(sentences=sentences)You might have issue with Laser3, we haven't properly tested it as we don't need it.Remove modelsIf you want to delete laser models they are here, run :rm-r$HOME/.cache/laser-modelsLicenseLASER is BSD-licensed, as found in theLICENSEfile in the root directory of this source tree.