Package: polyhymnia
polyhymniaPolyhymnia (/pɒliˈhɪmniə/; Greek: Πολυύμνια, lit. 'the one of many hymns'), alternatively Polymnia (Πολύμνια) was in Greek mythology the Muse of sacred poetry, sacred hymn, dance, and eloquence as well as agriculture and pantomime.Polyhymnia name comes from the Greek words "poly" meaning "many" and "hymnos", which means "praise".将praise理解为增强，那么该项目就是对“诗歌”的增强，即对自然语言数据的增广（我说是就是不接受反驳）。那么，该项目旨在给各位NLP工程师提供一些开箱即用的数据增广办法。Survey哈工大｜NLP数据增强方法？我有15种Installation推荐使用本地安装模式。本地安装：gitclonehttps://github.com/luoy2/polyhymnia.gitcdpolyhymnia
pipinstall.测试安装：~/polyhymnia$python38-munittesttests/test_methods.py-v
test_aeda(tests.test_methods.TestMethods)...[jieba]defaultdictfilepath../data/vocab.txt[jieba]defaultdictfilepath../data/vocab.txt[jieba]loaddefaultdict../data/vocab.txt...
/opt/python38/lib/python3.8/site-packages/pkg_resources/__init__.py:1151:DeprecationWarning:Useofinafuturerelease.returnget_provider(package_or_requirement).get_resource_stream([jieba]loaddefaultdict../data/vocab.txt...
>>Synonymsloadwordsegdict[/data/wanting/.local/lib/python3.8/site-packages/synonyms/data/vocab.t
>>Synonymsonloadingstopwords[/data/wanting/.local/lib/python3.8/site-packages/synonyms/data/stop
/data/wanting/.local/lib/python3.8/site-packages/synonyms/synonyms.py:104:ResourceWarning:unclosedsynonyms/data/stopwords.txt' mode='r' encoding='utf-8'>_load_stopwords(_fin_stopwords_path)ResourceWarning: Enable tracemalloc to get the object allocation traceback[Synonyms] on loading vectors [/data/comm/pkgs/polyhymnia/polyhymnia/data/words.vector] .../data/wanting/.local/lib/python3.8/site-packages/smart_open/smart_open_lib.py:479: DeprecationWarningopen/blob/develop/MIGRATING_FROM_OLDER_VERSIONS.rst for more informationwarnings.warn(message, category=DeprecationWarning)/data/wanting/.local/lib/python3.8/site-packages/synonyms/word2vec.py:175: DeprecationWarning: The biputs. Use frombuffer insteadweights = fromstring(fin.read(binary_len), dtype=REAL)['身份证丢了怎。么办', '身份证丢了怎。么办', '身份证丢了怎。么，\t办', '身份证丢了怎么办']oktest_eda (tests.test_methods.TestMethods) ... /data/wanting/.local/lib/python3.8/site-packages/scipy/e n to be a power of 2.warnings.warn("The balance properties of Sobol'pointsrequire"[Polyhnmnia] 2021-10-21 11:11:38,266 - DEBUG - execute tasks:[Polyhnmnia] 2021-10-21 11:11:38,266 - DEBUG - random_insertion: 3 times[Polyhnmnia] 2021-10-21 11:11:38,266 - DEBUG - synonym_replacement: 2 times[Polyhnmnia] 2021-10-21 11:11:38,266 - DEBUG - random_deletion: 2 times[Polyhnmnia] 2021-10-21 11:11:38,266 - DEBUG - random_swap: 2 times[Polyhnmnia] 2021-10-21 11:11:38,302 - DEBUG - random_insertion --- 谷物小麦种植[Polyhnmnia] 2021-10-21 11:11:38,341 - DEBUG - synonym_replacement --- 玉米栽植[Polyhnmnia] 2021-10-21 11:11:38,341 - DEBUG - random_deletion --- 小麦种植[Polyhnmnia] 2021-10-21 11:11:38,341 - DEBUG - random_swap --- 种植小麦[Polyhnmnia] 2021-10-21 11:11:38,341 - DEBUG - random_swap --- 种植小麦[Polyhnmnia] 2021-10-21 11:11:38,341 - DEBUG - random_deletion --- 小麦种植[Polyhnmnia] 2021-10-21 11:11:38,342 - DEBUG - synonym_replacement --- 小麦甘蔗[Polyhnmnia] 2021-10-21 11:11:38,342 - DEBUG - random_insertion --- 棉花小麦种植[Polyhnmnia] 2021-10-21 11:11:38,342 - DEBUG - random_insertion --- 农作物小麦种植['种植小麦', '小麦种植', '谷物小麦种植', '小麦种植', '种植小麦', '农作物小麦种植', '小麦甘蔗', '棉花小oktest_reverse_translate (tests.test_methods.TestMethods) ... 请使用 ReverseTranslate.set_creds(appid,oktest_simbert (tests.test_methods.TestMethods) ... 2021-10-21 11:11:38.407926: I tensorflow/stream_exebcudart.so.11.0['身份证丢了，怎么办','身份证丢了怎么办?','身份证丢了怎么办啊','身份证丢了怎么办？','身份证丢了怎么办！','身份证丢失怎么办？','身份证丢了该怎么办','身份证丢失，怎么办？']ok----------------------------------------------------------------------Ran 4 tests in 11.913sUsagesimbert 数据生成使用simbert_v2 (SimBERTv2来了！融合检索和生成的RoFormer-Sim模型) 进行相似句生成。frompolyhymniaimportSimbertSimbert.gen("身份证丢了怎么办",8)Out[38]:['身份证丢了怎么办?','身份证丢了怎么办？','身份证丢了怎么办。','身份证丢失了怎么办','身份证丢失了怎么办？','身份证丢了咋办？','身份证丢失怎么办？','身份证丢失怎么办！']默认会使用GPU进行生成，如果想使用CPU，请自行在引用包之前设置CUDA_VISIBLE_DEVICES 环境变量:importosos.environ['CUDA_VISIBLE_DEVICES']='-1'frompolyhymniaimportSimbertreverse_translate 多链翻译使用之前请先于https://api.fanyi.baidu.com/product/11申请翻译api appid和appsecretfrompolyhymniaimportReverseTranslateIn[3]:ReverseTranslate.set_creds(appid,appSecret)In[4]:ReverseTranslate.gen("小麦种植",4)[Polyhnmnia]2021-10-2018:05:04,331-DEBUG-starttranslatefor:小麦种植[Polyhnmnia]2021-10-2018:05:05,744-DEBUG-zh->hu->spa->zh:小麦栽培[Polyhnmnia]2021-10-2018:05:07,481-DEBUG-zh->dan->rom->zh:小麦种植；[Polyhnmnia]2021-10-2018:05:08,424-DEBUG-zh->bul->zh:小麦作物[Polyhnmnia]2021-10-2018:05:09,301-DEBUG-zh->en->zh:小麦种植Out[4]:['小麦种植','小麦作物','小麦种植；','小麦栽培']EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks在此进行了一些改进，使用sobol随机序列生成随机任务顺序，保证任务随机性In[1]:frompolyhymniaimportEDAIn[2]:EDA.gen("小麦种植",8)[Polyhnmnia]2021-10-2018:10:24,632-DEBUG-executetasks:[Polyhnmnia]2021-10-2018:10:24,632-DEBUG-random_insertion:3times[Polyhnmnia]2021-10-2018:10:24,632-DEBUG-synonym_replacement:2times[Polyhnmnia]2021-10-2018:10:24,632-DEBUG-random_deletion:2times[Polyhnmnia]2021-10-2018:10:24,632-DEBUG-random_swap:2times[Polyhnmnia]2021-10-2018:10:24,673-DEBUG-random_insertion---玉米小麦种植[Polyhnmnia]2021-10-2018:10:24,717-DEBUG-synonym_replacement---甜菜作物[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-random_deletion---小麦种植[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-random_swap---种植小麦[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-random_swap---种植小麦[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-random_deletion---小麦种植[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-synonym_replacement---大豆栽植[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-random_insertion---马铃薯小麦种植[Polyhnmnia]2021-10-2018:10:24,718-DEBUG-random_insertion---种植小麦种植Out[2]:['种植小麦','小麦种植','种植小麦种植','玉米小麦种植','小麦种植','种植小麦','甜菜作物', '大豆栽植', '马铃薯小麦种植']高级apipolyhymnia.methods.noising.stacking.eda.genalpha_sr同义词替换概率, 默认为0.1alpha_ri随机插入概率, 默认为0.1alpha_rs随机替换词顺序概率，默认为0.1p_rd随机删除概率，默认为0.1使用者可以使用高级api进行任务组合，如：frompolyhymnia.methods.noising.stacking.edaimportgencontents='小麦种植'print(gen(contents,num_aug=8,p_rd=0,alpha_ri=0))[Polyhnmnia] 2021-10-20 18:15:03,438 - DEBUG - execute tasks: 
[Polyhnmnia] 2021-10-20 18:15:03,438 - DEBUG - synonym_replacement: 4 times
[Polyhnmnia] 2021-10-20 18:15:03,438 - DEBUG - random_swap: 4 times
[Polyhnmnia] 2021-10-20 18:15:03,438 - DEBUG - synonym_replacement --- 小麦耕种
[Polyhnmnia] 2021-10-20 18:15:03,438 - DEBUG - synonym_replacement --- 小麦栽植
[Polyhnmnia] 2021-10-20 18:15:03,438 - DEBUG - random_swap --- 种植小麦
[Polyhnmnia] 2021-10-20 18:15:03,439 - DEBUG - random_swap --- 种植小麦
[Polyhnmnia] 2021-10-20 18:15:03,439 - DEBUG - synonym_replacement --- 小麦种植
[Polyhnmnia] 2021-10-20 18:15:03,439 - DEBUG - synonym_replacement --- 谷物种植
[Polyhnmnia] 2021-10-20 18:15:03,439 - DEBUG - random_swap --- 种植小麦
[Polyhnmnia] 2021-10-20 18:15:03,439 - DEBUG - random_swap --- 种植小麦

Out[6]: ['种植小麦', '小麦耕种', '种植小麦', '小麦栽植', '谷物种植', '小麦种植', '种植小麦', '种植小
麦']可以发现的是，当设置某些概率为0时，基于sobol序列的特性，EDA会均匀分配概率非0的任务进行数据增强组合。Ref:Sobol序列低差异序列AEDA: An Easier Data Augmentation Technique for Text ClassificationIn[6]:frompolyhymniaimportAEDAIn[7]:AEDA.gen("身份证丢了怎么办",4)Out[7]:['身份证丢了怎么,办','身份证丢了怎么办。','身 份证丢了怎么办','身份证丢了怎么办']高级apipolyhymnia.methods.noising.insertion.aeda_textfraction插入标点符号数量比例，默认为1/3puncs插入标点符号列表，默认为[",", "。", "，", "\t", " "]Logging使用polyhymnia._logger.LoggingFactory进行模块日志配置, 使用polyhymnia.set_verbose来进行快速切换日志等级。In[1]:importpolyhymniaIn[2]:polyhymnia.ReverseTranslate.set_creds(appid,appsecret)In[3]:polyhymnia.ReverseTranslate.gen("你是狗吗",3)Out[3]:['你是狗吗？','你是一只狗。']# 此时没有日志信息In[4]:importpolyhymniaIn[5]:polyhymnia.set_verbose(True)In[6]:ReverseTranslate.gen("你是狗吗",3)[Polyhnmnia]2021-10-2108:21:05,464-DEBUG-starttranslatefor:你是狗吗[Polyhnmnia]2021-10-2108:21:06,668-DEBUG-zh->el->en->zh:你是一只狗。[Polyhnmnia]2021-10-2108:21:08,065-DEBUG-zh->cs->ara->zh:你是狗吗？[Polyhnmnia]2021-10-2108:21:09,438-DEBUG-zh->est->pt->zh:你是狗吗？[Polyhnmnia]2021-10-2108:21:10,775-DEBUG-zh->bul->ru->zh:你是狗吗？[Polyhnmnia]2021-10-2108:21:11,693-DEBUG-zh->en->zh:你是狗吗Out[6]:['你是狗吗','你是狗吗？','你是一只狗。']鸣谢bert4kerasRoFormer-Sim: Integrating Retrieval and Generation into RoFormerchatopera/Synonyms: 中文近义词：聊天机器人，智能问答工具包fxsjy/jieba: 结巴中文分词tsroten/zhon: Constants used in Chinese text processingeasy to use retry decorator in python - GitHubSciPy library main repository - GitHubtensorflow/tensorflow: An Open Source Machine ... - GitHubpsf/requests: A simple, yet elegant, HTTP library. - GitHubGitHub - zhanlaoban/EDA_NLP_for_Chinese通用翻译API - 百度翻译开放平台