Package: convector
ConvectorIntroductionConvectoris a tool designed to facilitate the unification of conversational datasets into a consistent format. Capable of handling various dataset formats including JSONL, Parquet, Zstandard (Zst), JSON.GZ, CSV, and TXT,Convectorconverts them into JSONL format. The user can choose to filter data during transformation for enhanced customization. It offers flexibility in output formats with options like the default format and chat_completion format, the latter being compliant with OpenAI's format.InstallationConvectorcan be installed either via PyPI or directly from GitHub.Using PyPI: Runpip install convectorin the terminal.Using GitHub: Clone and install using the following commands:gitclonehttps://github.com/teilomillet/convectorcdconvector
pipinstall.UsageConvectorprovides a command-line interface for easy data processing with various customization options.Basic Command:convector process <file_path> [OPTIONS]Options:-p, --profile: Predefined profile from the config (default is 'default').-c, --conversation: Allow to process conversational exchanges.--instruction: Key for instructions or system messages.-i, --input-key: Key for user inputs.-o, --output-key: Key for bot responses.-s, --schema: Schema of the output data.--filter: Filter conditions in "field,operator,value" format.-l, --limit: Limit to a number of lines.--bytes: Limit to a number of bytes.-f, --file-out: File for transformed data.-d, --dir-out: Directory for output files.-v, --verbose: Enable detailed logs.Example Commands:Process each file in a folder:convectorprocess/path/to/data/Process the filedata.jsonl, which is a conversation-c, keep all the data with anidunder 10500, the output will be saved in/path/to/output_dir/output.jsonl:convectorprocess/data.jsonl-c--filterid<10500-foutput.jsonl-d/output_dir/Process the filedata.parquetand output the data into achat_completionformat with theidanduser_idat each row. (the output data will be saved indata_tr.jsonlinside the default output location (convector/silo)):convectorprocess/data.parquet--filter"id;user_id"--schemachat_completionRegister a profile namesampler, process333lines of the filedata.parquetand save the output into achat_completionformat in a file namesampler.jsonl:convectorprocess/data.json-psampler-l333-schat_completion-fsampler.jsonlProcess all the files in the folder/data, using all the commands previously saved in the profilesampler(see above):convectorprocess/data/-psamplerAdvanced FeaturesConversational Data Handling:Convectorefficiently processes nested conversational data. Using the--conversationcommand, it can identify and handle complex conversation structures, auto-generating aconversation_idwhen needed.Customization: Users can customize the data fields to be retained during processing with the--filteroption. By default,Convectorkeepsinstruction,input, andoutput. Additional fields can be included as required.Folder Handling:Convectorcan go through folders to process the data inside it. It will by default, create a file using_trat the end if no--file-outis specified.Configuration and CustomizationProfile Customization: Users can define and use custom profiles for different types of data processing tasks inside theconfig.yaml. The profile will automatically be saved and updated if used with new commands.Schema Application:Convectorallows for the application of custom schemas to tailor the output according to specific requirements.Default Schema:{"instruction":"","input":"","output":"","source":""}Chat_completion Schema:"messages":[{"role":"system","content":""},{"role":"user","content":""},{"role":"assistant","content":""}],"source":""