Package: charmr
charmra Magical AI File ConverterThis application will convert, transform, filter etc. any text based common file format into another file, based on a text description of the required result.It is using GPT-3.5/4 to generate code that will run the conversion locally. The actual input data is not being sent outside of the local machine.Getting Startedpip install charmrSet OpenAI API Keyexport OPENAI_API_KEY=<your open ai api key>Usageoptions:
  -h, --help            show this help message and exit
  --input_file INPUT_FILE, -i INPUT_FILE
                        Input file
  --output_file OUTPUT_FILE, -o OUTPUT_FILE
                        Output file
  --conversion CONVERSION, -c CONVERSION
                        Conversion description
  --alias ALIAS, -a ALIAS
                        Conversion alias, for reuse
  --view_code_only, -v  View code without running
  --gpt_4, -4           Use GPT-4
  --include_input_rows INCLUDE_INPUT_ROWS, -r INCLUDE_INPUT_ROWS
                        Include N input rows in promptDefine--input_file,--output_fileand--conversionfor basic functionality.Can also use stdin and/or stdout instead of defining in/out path.Adding an--aliaswill save the conversion in a.charmrfile.Then you can specify--aliaswithout providing--conversionto repeat the same conversion.Define--view_code_onlywill only print out the code without running it. Use this to review generated code for safety. Specifying--aliaswill save it to be reused.Use--gpt_4to use GPT-4 to generate code. Default isgpt-3.5-turboSpecifyinclude_input_rowswith number of rows to include in prompt. This can improve code generation.e.g.charmr -i input.csv -o output.json -c "convert csv to json"ORcharmr -c "convert csv to json" < input.csv > output.jsonOR a more fluent usecat input.csv | charmr -4 -c "convert the csv to json and keep just the first and third columns" | jq .
[
  {
    "a": "1",
    "c": "3"
  },
  {
    "a": "4",
    "c": "6"
  }
]Emulating thecututility:cat input.csv | cut -d"," -f2Same thing withcharmrcat input.csv | charmr -4 -c "read the csv and keep only the second column"Then combining transformation and filteringcat input.csv | charmr -4 -c "read the csv, remove the header row, keep only the second column and output the square of the values"ExamplesBasic conversioninputcsva, b, c
1, 2, 3
4, 5, 6conversionconvert csv to jsonoutputjson[{"a":"1"," b":" 2"," c":" 3"},{"a":"4"," b":" 5"," c":" 6"}]Column manipulationinputcsva, b, c
1, 2, 3
4, 5, 6conversionthe input file is a CSV with column headers a,b,c. the output file is a CSV. create a new column ac with the multiplication of a and coutputcsva,b,c,ac
1,2,3,3
4,5,6,24Conversion Logicinputcsvday,month,year
11,3,2021
2,8,1999conversioninput a csv file with day,month,year column headers. output a JSON file with an extra field which is the unix timestamp for the date represented in each row"outputjson[{"day":"11","month":"3","year":"2021","timestamp":1615449600},{"day":"2","month":"8","year":"1999","timestamp":933577200}]DisclaimerThe success of each conversion depends on GPT-3.5 generating the correct code for the task.
It also depends on the conversion string correctly describing the input and the logic of manipulating it.Security / SafetyInput file data never leaves your local machine. Only the conversion description is sent to OpenAI to generate the conversion code.Note that using the '--include_input_rows N' optionwillsend the first N rows of the input file to GPT to help with code generation accuracy