Package: lilac
LilacBetter data, better AI🔗Try the Lilac web demo!Lilac is a tool for exploration, curation and quality control of datasets for training, fine-tuning
and monitoring LLMs.Lilac is used by companies likeCohereandDatabricksto visualize, quantify and improve the quality of
pre-training and fine-tuning data.Lilac runson-deviceusing open-source LLMs with a UI and Python API.🆒 NewLilac Gardenis our hosted platform for blazing fast
dataset-level computations.Sign upto join the pilot.Cluster & title millions of documents with the power of LLMs.Explore and searchover 36,000 clusters of 4.3M documents in OpenOrcaWhy use Lilac?Explore your data interactively with LLM-powered search, filter, clustering and annotation.Curate AI data, applying best practices like removing duplicates, PII and obscure content to
reduce dataset size and lower training cost and time.Inspect and collaborate with your team on a single, centralized dataset to improve data quality.Understand how data changes over time.Lilac can offload expensive computations toLilac Garden, our
hosted platform for blazing fast dataset-level computations.See our3min walkthrough video🔥 Getting started💻 Installpipinstalllilac[all]If you prefer no local installation, you can duplicate ourSpaces demoby following documentationhere.For more detailed instructions, see ourinstallation guide.🌐 Start a webserverStart a Lilac webserver with ourlilacCLI:lilacstart~/my_projectOr start the Lilac webserver from Python:importlilacasllll.start_server(project_dir='~/my_project')This will open start a webserver athttp://localhost:5432/where you can now load datasets and
explore them.Lilac GardenLilac Garden is our hosted platform for running dataset-level computations. We utilize powerful GPUs
to accelerate expensive signals like Clustering, Embedding, and PII.Sign upto join the pilot.Cluster and titlea milliondata points in20 minsEmbed your dataset athalf a billiontokens per minRun your own signal📊 Load dataDatasets can be loaded directly from HuggingFace, Parquet, CSV, JSON,LangSmith from LangChain, SQLite,LLamaHub, Pandas, Parquet, and more. More documentationhere.importlilacasllll.set_project_dir('~/my_project')dataset=ll.from_huggingface('imdb')If you prefer, you can load datasets directly from the UI without writing any Python:🔎 Explore[!NOTE]
🔗 ExploreOpenOrcaandits clustersbefore installing!Once we've loaded a dataset, we can explore it from the UI and get a sense for what's in the data.
More documentationhere.✨ ClusteringCluster any text column to get automated dataset insights:dataset=ll.get_dataset('local','imdb')dataset.cluster('text')# add `use_garden=True` to offload to Lilac Garden[!TIP]
Clustering on device can be slow or impractical, especially on machines without a powerful GPU or
large memory. Offloading the compute toLilac Garden, our
hosted data processing platform, can speedup clustering by more than 100x.⚡ Annotate with Signals (PII, Text Statistics, Language Detection, Neardup, etc)Annotating data with signals will produce another column in your data.dataset=ll.get_dataset('local','imdb')dataset.compute_signal(ll.LangDetectionSignal(),'text')# Detect language of each doc.# [PII] Find emails, phone numbers, ip addresses, and secrets.dataset.compute_signal(ll.PIISignal(),'text')# [Text Statistics] Compute readability scores, number of chars, TTR, non-ascii chars, etc.dataset.compute_signal(ll.PIISignal(),'text')# [Near Duplicates] Computes clusters based on minhash LSH.dataset.compute_signal(ll.NearDuplicateSignal(),'text')# Print the resulting manifest, with the new field added.print(dataset.manifest())We can also compute signals from the UI:🔎 SearchSemantic and conceptual search requires computing an embedding first:dataset.compute_embedding('gte-small',path='text')Semantic searchIn the UI, we can search by semantic similarity or by classic keyword search to find chunks of
documents similar to a query:We can run the same search in Python:rows=dataset.select_rows(columns=['text','label'],searches=[ll.SemanticSearch(path='text',embedding='gte-small')],limit=1)print(list(rows))Conceptual searchConceptual search is a much more controllable and powerful version of semantic search, where
"concepts" can be taught to Lilac by providing positive and negative examples of that concept.Lilac provides a set of built-in concepts, but you can create your own for very specifWe can create a concept in Python with a few examples, and search by it:concept_db=ll.DiskConceptDB()db.create(namespace='local',name='spam')# Add examples of spam and not-spam.db.edit('local','spam',ll.concepts.ConceptUpdate(insert=[ll.concepts.ExampleIn(label=False,text='This is normal text.'),ll.concepts.ExampleIn(label=True,text='asdgasdgkasd;lkgajsdl'),ll.concepts.ExampleIn(label=True,text='11757578jfdjja')]))# Search by the spam concept.rows=dataset.select_rows(columns=['text','label'],searches=[ll.ConceptSearch(path='text',concept_namespace='lilac',concept_name='spam',embedding='gte-small')],limit=1)print(list(rows))🏷️ LabelingLilac allows you to label individual points, or slices of data:We can also label all data given a filter. In this case, adding the label "short" to all text with a
small amount of characters. This field was produced by the automatictext_statisticssignal.We can do the same in Python:dataset.add_labels('short',filters=[(('text','text_statistics','num_characters'),'less',1000)])Labels can be exported for downstream tasks. Detailed documentationhere.💬 ContactFor bugs and feature requests, pleasefile an issue on GitHub.For general questions, pleasevisit our Discord.