Package: lm-buddy
LM Buddy[!WARNING]LM Buddy is in the early stages of development.
It is missing important features and documentation.
You should expect breaking changes in the core interfaces and configuration structures
as development continues.
Use only if you are comfortable working in this environment.LM Buddy is a collection of jobs for finetuning and evaluating open-source (large) language models.
The library makes use of YAML-based configuration files as inputs to CLI commands for each job,
and tracks input/output artifacts onWeights & Biases.The package currently exposes two types of jobs:finetuning jobusing HuggingFace model/training implementations andRay Trainfor compute scaling, or anevaluation jobusinglm-evaluation-harnesswith inference performed via an in-process HuggingFace model or an externally-hostedvLLMserver.InstallationLM Buddy is available on PyPI and can be installed as follows:pip install lm-buddyMinimum Python versionLM Buddy is intended to be used in production on a Ray cluster
(see section below onRay job submission).
Currently, we are utilizing Ray clusters running Python 3.10.8.
In order to avoid dependency/syntax errors when executing LM Buddy on Ray,
installation of this package requires Python between[3.10, 3.11).CLI usageLM Buddy exposes a CLI with a few commands, one for each type of job.
To see all available job commands, runlm_buddy run --helpOnce LM Buddy is installed in your local Python environment, usage is as follows:# Simple test
lm_buddy run simple --config simple_config.yaml

# LLM finetuning
lm_buddy run finetuning --config finetuning_config.yaml

# LLM evaluation
lm_buddy run lm-harness --config lm_harness_config.yamlSee theexamples/configsfolder for examples of the job configuration structure.
For a full end-to-end interactive workflow for using the package, see the example notebooks.Ray job submissionAlthough the LM Buddy CLI can be used as a standalone tool,
its commands are intended to be used as the entrypoints for jobs on aRaycompute cluster.
The suggested method for submitting an LM Buddy job to Ray is by using theRay Python SDKwithin a local Python driver script.
This requires you to specify a Ray runtime environment containing:Aworking_dirfor the local directory containing your job config YAML file, andApipdependency for your desired version oflm-buddy.Additionally, if your job requires GPU resources on the Ray entrypoint worker (e.g., for loading large/quantized models),
you should specify theentrypoint_num_gpusparameter upon submission.An example of the submission process is as follows:from ray.job_submission import JobSubmissionClient

# If using a remote cluster, replace 127.0.0.1 with the head node's IP address.
client = JobSubmissionClient("http://127.0.0.1:8265")

runtime_env = {
    "working_dir": "/path/to/working/directory",
    "pip": ["lm-buddy==X.X.X"]
    
}

# Assuming 'config.yaml' is present in the working directory
client.submit_job(
    entrypoint="lm_buddy run <job-name> --config config.yaml", 
    runtime_env=runtime_env,
    entrypoint_num_gpus=1
)See theexamples/folder for more examples of submitting Ray jobs.DevelopmentSee thecontributingguide for more information on development workflows
and/or building locally.