Package: laser-encoders
LASER encodersLASER Language-Agnostic SEntence Representations Toolkitlaser_encoders is the official Python package for the FacebookLASERlibrary. It provides a simple and convenient way to use LASER embeddings in Python. It allows you to calculate multilingual sentence embeddings using the LASER toolkit. These embeddings can be utilized for various natural language processing tasks, including document classification, bitext filtering, and mining.DependenciesPython>= 3.8PyTorch>= 1.10.0sacremoses>=0.1.0sentencepiece>=0.1.99numpy>=1.21.3fairseq>=0.12.2You can find a full list of requirementshereInstallationYou can installlaser_encoderspackage from PyPI:pipinstalllaser_encodersAlternatively, you can install it from a local clone of this repository, in editable mode:pipinstall.-eUsageHere's a simple example on how to obtain embeddings for sentences using theLaserEncoderPipeline:Note:By default, the models will be downloaded to the~/.cache/laser_encodersdirectory. To specify a different download location, you can provide the argumentmodel_dir=path/to/model/directoryfromlaser_encodersimportLaserEncoderPipeline# Initialize the LASER encoder pipelineencoder=LaserEncoderPipeline(lang="igbo")# Encode sentences into embeddingsembeddings=encoder.encode_sentences(["nnọọ, kedu ka ị mere"])# If you want the output embeddings to be L2-normalized, set normalize_embeddings to Truenormalized_embeddings=encoder.encode_sentences(["nnọọ, kedu ka ị mere"],normalize_embeddings=True)If you prefer more control over the tokenization and encoding process, you can initialize the tokenizer and encoder separately:fromlaser_encodersimportinitialize_encoder,initialize_tokenizer# Initialize the LASER tokenizertokenizer=initialize_tokenizer(lang="igbo")tokenized_sentence=tokenizer.tokenize("nnọọ, kedu ka ị mere")# Initialize the LASER sentence encoderencoder=initialize_encoder(lang="igbo")# Encode tokenized sentences into embeddingsembeddings=encoder.encode_sentences([tokenized_sentence])By default, thespmflag is set toTruewhen initializing the encoder, ensuring the accompanying spm model is downloaded.Supported Languages:You can specify any language from theFLORES200dataset. This includes both languages identified by their full codes (like "ibo_Latn") and simpler alternatives (like "igbo").Downloading the pre-trained modelsIf you prefer to download the models individually, you can use the following command:python-mlaser_encoders.download_models--lang=your_prefered_language# e.g., --lang="igbo""By default, the downloaded models will be stored in the~/.cache/laser_encodersdirectory. To specify a different download location, utilize the following command:python-mlaser_encoders.download_models--model-dir=path/to/model/directoryFor a comprehensive list of available arguments, you can use the--helpcommand with the download_models script.Once you have successfully downloaded the models, you can utilize theSentenceEncoderto tokenize and encode your text in your desired language. Here's an example of how you can achieve this:fromlaser_encoders.modelsimportSentenceEncoderfrompathlibimportPathencoder=SentenceEncoder(model_path=path/to/downloaded/model,spm_model=Path(path/to/spm_model),spm_vocab=path/to/cvocab)embeddings=encoder("This is a test sentence.")If you want to perform tokenization seperately, you can do this below:fromlaser_encoders.laser_tokenizerimportLaserTokenizertokenizer=LaserTokenizer(spm_model=Path(path/to/spm_model))tokenized_sentence=tokenizer.tokenize("This is a test sentence.")encoder=SentenceEncoder(model_path=path/to/downloaded/model,spm_vocab=path/to/cvocab)embeddings=encoder.encode_sentences([tokenized_sentence])For tokenizing a file instead of a string, you can use the following:tokenized_sentence=tokenizer.tokenize_file(inp_fname=Path(path/to/input_file.txt),out_fname=Path(path/to/output_file.txt))Now you can use these embeddings for downstream tasksFor more advanced usage and options, please refer to the official LASER repository documentation.LASER Versions and Associated PackagesFor users familiar with the earlier version of LASER, you might have encountered thelaserembeddingspackage. This package primarily dealt with LASER-1 model embeddings.For the latest LASER-2,3 models, use the newly introducedlaser_encoderspackage, which offers better performance and support for a wider range of languages.ContributingWe welcome contributions from the developer community to enhance and improve laser_encoders. If you'd like to contribute, you can:Submit bug reports or feature requests through GitHub issues.Fork the repository, make changes, and submit pull requests for review.Please follow ourContribution Guidelinesto ensure a smooth process.Code of ConductWe expect all contributors to adhere to ourCode of Conduct.ContributorsThe following people have contributed to this project:Victor JosephPaul OkewunmiSiddharth Singh RanaDavid DaleHolger SchwenkKevin HeffernanLicenseThis package is released under theLASERBSD License.