Package: promptcraft
PromptCraftA Prompt Perturbation Toolkit for Prompt Robustness AnalysisTable of ContentsInstallationCharacter EditingCharacter ReplacementCharacter DeletionCharacter InsertionCharacter SwapKeyboard TyposOptical Character Recognition (OCR)Word ManipulationSynonym ReplacementWord InsertionWord SwapWord DeletionInsert PunctuationWord SplitSentence ParaphrasingBack Translation based on 🤗 Hugging Face MarianMTModelBack Translation based on Google TranslatorParaphrasingFormal StyleCasual StylePassive StyleActive StyleParallel ProcessingStructure of the CodeCitationAcknowledgementInstallationpipinstallpromptcraftCharacter EditingCharacter-level Prompt PerturbationCharacterPerturbclass for manipulating character in a sentencefrompromptcraftimportcharactersentence="Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May."level=0.25# Percentage of characters that will be editedcharacter_tool=character.CharacterPerturb(sentence=sentence,level=level)Character ReplacementRandomly replacelevelpercentage characters from the sentencechar_replace=character_tool.character_replacement()Character DeletionRandomly deletelevelpercentage characters from the sentencechar_delete=character_tool.character_deletion()Character InsertionRandomly insertlevelpercentage characters to the sentencechar_insert=character_tool.character_insertion()Character SwapRandomly swaplevelpercentage characters in the sentenceNOTE: including self-swappingchar_swap=character_tool.character_swap()Keyboard TyposRandomly substitutelevelpercentage characters in the sentence
with a randomly chosen character which is near the original character in the Keyboard (USA Full-size Layout)NOTE:(1) We appliedkeyboard_distance=1, i.e., the nearest character, number, or samples.(2) If it is a character, we randomly chose lowercase or uppercase.char_keyboard=character_tool.keyboard_typos()Optical Character RecognitionRandomly substitutelevelpercentage characters in the sentence with a common OCR map errorchar_ocr=character_tool.optical_character_recognition()Word ManipulationWord-level Prompt PerturbationWordPerturbclass for manipulating words in a sentenceNOTE: the number of words in a sentence is only the valid words without considering spaces, special symbols, and punctuationsfrompromptcraftimportwordsentence="Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May."level=0.25# Percentage of words that will be manipulatedword_tool=word.WordPerturb(sentence=sentence,level=level)Synonym ReplacementRandomly choose $n$ words from the sentence that are not stop words.Replace each of these words with one of its synonyms chosen at random.Problem 1: Without any synonymsProblem 2: Fewer positions than needed positionsword_synonym=word_tool.synonym_replacement()Word InsertionFind a random synonym of a random word in the sentence that is not a stop word.Insert that synonym into a random position in the sentence.Do this $n$ times.word_insert=word_tool.word_insertion()Word SwapRandomly choose two words in the sentence and swap their positions.Do this $n$ times.word_swap=word_tool.word_swap()Word DeletionEach word in the sentence can be randomly removed with probability $p$.word_delete=word_tool.word_deletion()Insert PunctuationRandomly insert punctuation in the sentence with probability $p$.word_punctuation=word_tool.insert_punctuation()Word SplitRandomly split a word to two tokens randomlyword_split=word_tool.word_split()Sentence ParaphrasingSentence-level Prompt PerturbationSentencePerturbclass for directly manipulating a sentencefrompromptcraftimportsentencesen="Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May."sentence_tool=sentence.SentencePerturb(sentence=sen)Back Translation by Hugging FaceBack translate the sentence (English $\rightarrow$ German $\rightarrow$ English) via 🤗 Hugging Face MarianMTModelback_trans_hf=sentence_tool.back_translation_hugging_face()Back Translation by Google TranslatorBack translate the sentence (English $\rightarrow$ German $\rightarrow$ English) via Google Translate APIback_trans_google=sentence_tool.back_translation_google()ParaphrasingParaphrasing the sentence viaParrot Paraphraserconsidering(1)Adequency: Is the meaning preserved adequately?(2)Fluency: Is the paraphrase fluent English?(3)Diversity: (Lexical / Phrasal / Syntactical): How much has the paraphrase changed the original sentence?sen_paraphrase=sentence_tool.paraphrase()Formal StyleTransform the sentence style to Formalsen_formal=sentence_tool.formal()Casual StyleTransform the sentence style to Casualsen_casual=sentence_tool.casual()Passive StyleTransform the sentence style to Passivesen_passive=sentence_tool.passive()Active StyleTransform the sentence style to Activesen_active=sentence_tool.active()Parallel ProcessingSince all the methods are executed on the CPU,
they can be performed in parallel using themultiprocessingpackage.Structure of the CodeAt the root of the project, you will see:.
├── LICENSE
├── README.md
├── promptcraft
│   ├── __init__.py
│   ├── character.py
│   ├── parrot.py
│   ├── sentence.py
│   ├── styleformer.py
│   └── word.py
├── setup.cfg
└── setup.pyCitationIf you find our toolkit useful, please consider citing our repo and toolkit in your publications. We provide a BibTeX entry below.@misc{JiaPromptCraft23,author={Jia, Shuyue},title={{PromptCraft}: A Prompt Perturbation Toolkit},year={2023},publisher={GitHub},journal={GitHub Repository},howpublished={\url{https://github.com/SuperBruceJia/promptcraft}},}@misc{JiaAwesomeLLM23,author={Jia, Shuyue},title={Awesome {LLM} Self-Consistency},year={2023},publisher={GitHub},journal={GitHub Repository},howpublished={\url{https://github.com/SuperBruceJia/Awesome-LLM-Self-Consistency}},}@misc{JiaAwesomeSTS23,author={Jia, Shuyue},title={Awesome Semantic Textual Similarity},year={2023},publisher={GitHub},journal={GitHub Repository},howpublished={\url{https://github.com/SuperBruceJia/Awesome-Semantic-Textual-Similarity}},}AcknowledgementThis work was finished during my 2023 fall semester research rotation
at the Department of Electrical and Computer Engineering, Boston University.