AnarchyAI/normalizer

Objective

Normalizer is a tool that processes various file types to extract prompt-response pairs for finetuning LLM models on unstructured data.

The end-state will be a GUI where users of any technical level can drag and drop multiple files at once into the application, then they hit a 'create fine-tuning data' button and they'll receive a .csv with a system prompt and a series of matched pair prompt-completion responses.

Working Repository

https://github.com/anarchy-ai/normalizer

Input Formats

• Text: .txt, .md, .mdx
• Documents: .pdf, .doc, .docx
• Spreadsheets: .xlsx, .csv
• Code: .py, .js, .html, .java
• Images: .jpg, .png (will require OCR)

Output Format

A .csv file with 3 columns:

System Prompt	User Prompt	Response
Hi, how can I help you today?
	What do these lab results suggest?	These lab results suggest that the patient is healthy, as no anomalous data has been detected.
	What is the sentiment of the last 5 customers who came into support chat?	The last five customers have a neutral to positive sentiment.
	...	...

Required Libraries

pip install PyPDF2 python-docx pandas openpyxl pillow pytesseract beautifulsoup4 transformers datasets

Project Structure

normalizer/
│
├── src/
│   ├── __init__.py
│   ├── file_ingest.py
│   ├── prompt_extractor.py
│   ├── main.py
│
├── app.py
├── requirements.txt
├── README.md
└── .gitignore