Package: lm-scorer
lm-scorer📃 Language Model based sentences scoring librarySynopsisThis package provides a simple programming interface to score sentences using different MLlanguage models.A simpleCLIis also available for quick prototyping.You can run it locally or on directly on Colab usingthis notebook.Do you believe that this isuseful?
Has itsaved you time?
Or maybe you simplylike it?If so,support this work with a Star ⭐️.Installpipinstalllm-scorerUsageimporttorchfromlm_scorer.models.autoimportAutoLMScorerasLMScorer# Available modelslist(LMScorer.supported_model_names())# => ["gpt2", "gpt2-medium", "gpt2-large", "gpt2-xl", distilgpt2"]# Load model to cpu or cudadevice="cuda:0"iftorch.cuda.is_available()else"cpu"batch_size=1scorer=LMScorer.from_pretrained("gpt2",device=device,batch_size=batch_size)# Return token probabilities (provide log=True to return log probabilities)scorer.tokens_score("I like this package.")# => (scores, ids, tokens)# scores = [0.018321, 0.0066431, 0.080633, 0.00060745, 0.27772, 0.0036381]# ids    = [40,       588,       428,      5301,       13,      50256]# tokens = ["I",      "Ġlike",   "Ġthis",  "Ġpackage", ".",     "
"]# Compute sentence score as the product of tokens' probabilitiesscorer.sentence_score("I like this package.",reduce="prod")# => 6.0231e-12# Compute sentence score as the mean of tokens' probabilitiesscorer.sentence_score("I like this package.",reduce="mean")# => 0.064593# Compute sentence score as the geometric mean of tokens' probabilitiesscorer.sentence_score("I like this package.",reduce="gmean")# => 0.013489# Compute sentence score as the harmonic mean of tokens' probabilitiesscorer.sentence_score("I like this package.",reduce="hmean")# => 0.0028008# Get the log of the sentence score.scorer.sentence_score("I like this package.",log=True)# => -25.835# Score multiple sentences.scorer.sentence_score(["Sentence 1","Sentence 2"])# => [1.1508e-11, 5.6645e-12]# NB: Computations are done in log space so they should be numerically stable.CLIThe pip package includes a CLI that you can use to score sentences.usage: lm-scorer [-h] [--model-name MODEL_NAME] [--tokens] [--log-prob]
                 [--reduce REDUCE] [--batch-size BATCH_SIZE]
                 [--significant-figures SIGNIFICANT_FIGURES] [--cuda CUDA]
                 [--debug]
                 sentences-file-path

Get sentences probability using a language model.

positional arguments:
  sentences-file-path   A file containing sentences to score, one per line. If
                        - is given as filename it reads from stdin instead.

optional arguments:
  -h, --help            show this help message and exit
  --model-name MODEL_NAME, -m MODEL_NAME
                        The pretrained language model to use. Can be one of:
                        gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2.
  --tokens, -t          If provided it provides the probability of each token
                        of each sentence.
  --log-prob, -lp       If provided log probabilities are returned instead.
  --reduce REDUCE, -r REDUCE
                        Reduce strategy applied on token probabilities to get
                        the sentence score. Available strategies are: prod,
                        mean, gmean, hmean.
  --batch-size BATCH_SIZE, -b BATCH_SIZE
                        Number of sentences to process in parallel.
  --significant-figures SIGNIFICANT_FIGURES, -sf SIGNIFICANT_FIGURES
                        Number of significant figures to use when printing
                        numbers.
  --cuda CUDA           If provided it runs the model on the given cuda
                        device.
  --debug               If provided it provides additional logging in case of
                        errors.DevelopmentYou can install this library locally for development using the commands below.
If you don't have it already, you need to installpoetryfirst.# Clone the repogitclonehttps://github.com/simonepri/lm-scorer# CD into the created foldercdlm-scorer# Create a virtualenv and install the required dependencies using poetrypoetryinstallYou can then run commands inside the virtualenv by usingpoetry run COMMAND.Alternatively, you can open a shell inside the virtualenv usingpoetry shell.If you wish to contribute to this project, run the following commands locally before opening a PR and check that no error is reported (warnings are fine).# Run the code formatterpoetryruntaskformat# Run the linterpoetryruntasklint# Run the static type checkerpoetryruntasktypes# Run the testspoetryruntasktestAuthorsSimone Primarosa-simonepriSee also the list ofcontributorswho participated in this project.LicenseThis project is licensed under the MIT License - see thelicensefile for details.