Package: llm-dataset-converter
Thellm-dataset-converterallows the conversion between
various dataset formats for large language models (LLMs).
Filters can be supplied as well, e.g., for cleaning up the data.Dataset formats:pairs: alpaca (r/w), csv (r/w), jsonl (r/w), parquet (r/w), tsv (r/w), xtuner (r/w)pretrain: csv (r/w), jsonl (r/w), parquet (r/w), tsv (r/w), txt (r/w)translation: csv (r/w), jsonl (r/w), parquet (r/w), tsv (r/w), txt (r/w)classification: csv (r/w), jsonl (r/w), parquet (r/w), tsv (r/w)Compression formats:bzipgzipxzzstdExamples:Simple conversion with logging info:llm-convert \
  from-alpaca \
    -l INFO \
    --input ./alpaca_data_cleaned.json \
  to-csv-pr \
    -l INFO \
    --output alpaca_data_cleaned.csvAutomatic decompression/compression (based on file extension):llm-convert \
  from-alpaca \
    --input ./alpaca_data_cleaned.json.xz \
  to-csv-pr \
    --output alpaca_data_cleaned.csv.gzFiltering:llm-convert \
  -l INFO \
  from-alpaca \
    -l INFO \
    --input alpaca_data_cleaned.json \
  keyword \
    -l INFO \
    --keyword function \
    --location any \
    --action keep \
  to-alpaca \
    -l INFO \
    --output alpaca_data_cleaned-filtered.jsonChangelog0.2.0 (2024-02-27)added support for XTuner conversation JSON format:from-xtunerandto-xtuneradded filterupdate-pair-datato allow tweaking or rearranging of the dataintroducedldc.apimodule to separate out abstract superclasses and avoid circular importsreaders now set the ‘file’ meta-data valueaddedfile-filterfilter for explicitly allowing/discarding records that stem from certain files (entry in meta-data: ‘file’)addedrecord-filesfilter for recording the files that the records are based on (entry in meta-data: ‘file’)filterpretrain-sentences-to-pairscan now omit filling theinstructionwhen using 0 as prompt steprequiring seppl>=0.1.2 nowadded global option-U, –unescape_unicodetollm-converttool to allow conversion of escaped unicode charactersthellm-appendtool now supports appending for json, jsonlines and CSV files apart from plain-text files (default)0.1.1 (2024-02-15)addedclassificationdomainaddedfrom-jsonlines-clreader andto-jsonlines-clwriter for classification data in JSON lines formatadded filterpretrain-sentences-to-classificationto turn pretrain data into classification data (with a predefined label)added filterclassification-label-mapthat can generate a label string/int maptheto-llama2-formatfilter now has the–skip_tokensoptions to leave out the 
 
 tokensaddedfrom-parquet-clreader andto-parquet-clwriter for classification data in Parquet database formataddedfrom-csv-cl/from-tsv-clreaders andto-csv-cl/to-tsv-clwriters for classification data in CSV/TSV file format0.1.0 (2024-02-05)fixed output format ofto-llama2-formatfilterllama2-to-pairsfilter has more robust parsing nowupgraded seppl to 0.1.0switched to seppl classes: Splitter, MetaDataHandler, Reader, Writer, StreamWriter, BatchWriter0.0.5 (2024-01-24)added flag-b/–force_batchto thellm-converttool which all data to be reader from the
reader before filtering it and then passing it to the writer; useful for batch filters.added therandomize-recordsbatch filteradded the–encoding ENCoption to file readersauto-determined encoding is now being logged (INFOlevel)theLDC_ENCODING_MAX_CHECK_LENGTHenvironment variable allows overriding the default
number of bytes used for determining the file encoding in auto-detect modedefault max number of bytes inspected for determining file encoding is now 10kbmethodlocate_filesinbase_iono longer includes directories when expanding globsadded toolllm-file-encodingfor determining file encodings of text filesadded methodreplace_extensiontobase_iomodule for changing a files extension
(removes any supported compression suffix first)stream writers (.jsonl/.txt) now work with–force_batchmode; the output file name
gets automatically generated from the input file name when just using a directory for
the output0.0.4 (2023-12-19)pairs-to-llama2filter now has an optional–prefixparameter to use with the instructionadded thepretrain-sentences-to-pairsfilter for generating artificial prompt/response datasets from pretrain datarequires seppl>=0.0.11 nowtheLDC_MODULES_EXCLenvironment variable is now used for specifying modules to be excluded from the registration
process (e.g., used when generating help screens for derived libraries that shouldn’t output the
base plugins as well)llm-registryandllm-helpnow allow specifying excluded modules via-e/–excluded_modulesoptionto-alpacawriter now has the-a/–ensure_asciiflag to enforce ASCII compatibility in the outputadded global option-u/–update_intervaltoconverttool to customize how often progress of # records
processed is being output in the console (default: 1000)text-lengthfilter now handles None values, i.e., ignores themlocations (i.e., input/instructions/output/etc) can be specified now multiple timesthellm-helptool can generate index files for all the plugins now; in case of markdown
it will link to the other markdown files0.0.3 (2023-11-10)added therecord-windowfilteradded thellm-registrytool for querying the registry from the command-lineadded thereplace_patternsmethod toldc.text_utilsmoduleadded thereplace-patternsfilteradded-p/–pretty-printflag toto-alpacawriteraddedpairs-to-llama2andllama2-to-pairsfilter
(since llama2 has instruction as part of the string, it is treated as pretrain data)addedto-llama2-formatfilter for pretrain records (no 
…
 block)now requiring seppl>=0.0.8 in order to raise Exceptions when encountering unknown arguments0.0.2 (2023-10-31)addedtext-statsfilterstream writers accept iterable of data records now as well to improve throughputtext_utils.apply_max_lengthnow uses simple whitespace splitting instead of
searching for nearest word boundary to break a line, which results in a massive
speed improvementfix:text_utils.remove_patternsno longer multiplies the generated lines when using
more than one patternaddedremove-patternsfilterpretrain and translation text writers now buffer records by default (-b,–buffer_size)
in order to improve throughputjsonlines writers for pair, pretrain and translation data are now stream writers0.0.1 (2023-10-26)initial release