Saravanan666/pypi-org huggingface dataset:
Package: textColor
This is an easy to use Python library which allows you to make your terminal outputs more colorful and therefore easier to read and understand.It currently only works with Python3The creator and maintainer is Jannik Ramrath (jramrath)GitHub:https://github.com/jramrath/textColorPyPi:https://pypi.org/project/textColor/ContributingIf you've found a bug or a typo, feel free to open an Issue on GitHub. Already have a solution? Make a Pull request and I'll take a look at your changes. Please make sure a similar Issue/Pull request doesn't already exist.InstallationThis library is hosted onpypi.org. You can install it withpip:pip install textColorUsageTo use this library, you first have to import it:import textColorTo print colored text do the following:print(textColor.green("GREEN"))           # 'GREEN' will be green
print(textColor.red("RED"))               # 'RED' will be red
print(textColor.blue("BLUE"))             # 'BLUE' will be blue
print(textColor.yellow("YELLOW"))         # 'YELLOW' will be yellowYou can also use special 'pre-fixes':print(textColor.prompt("Name: "))         # will output '[?] Name: '  while '[?]' is yellow
print(textColor.info("Info"))             # will output '[-] Info'    while '[-]' is blue
print(textColor.error("ERROR"))           # will output '[!] ERROR'   while '[!]' is red
print(textColor.output("Success"))        # will output '[>] Success' while '[>]' is greenIf you dont' want to type input()/print() you can use the 'print functions':textColor.pPromt("Name: ")                # will call input("[?] Name: ") and return
                                          #  the output, while '[?]’ is yellow
textColor.pInfo("Info")                   # will output '[-] Info'    while '[-]' is blue
textColor.pError("Error")                 # will output '[!] ERROR'   while '[!]' is red
textColor.pOutput("Success")              # will output '[>] Success' while '[>]' is greenIf those function names seem too long, you should use abbreviations:green   ––> g
red     ––> r
blue    ––> b
yellow  ––> y

error   ––> err
output  ––> out

pError  ––> pErr
pOutput ––> pOutLicenseThis library was published under theGNU General Public License. For more information take a look at theLICENSEfile.
Package: text-comparator
Text Comparator PackageDescriÃ§Ã£o.O pacote Text Comparator Package Ã© usado para a avaliaÃ§Ã£o de similaridade entre textos, com base nas seguintes estatÃ­sticas:
Tamanho mÃ©dio de palavra: MÃ©dia simples do nÃºmero de caracteres por palavra.RelaÃ§Ã£o Type-Token:NÃºmero de palavras diferentes utilizadas em um texto divididas pelo total de palavras.RazÃ£o Hapax Legomana:NÃºmero de palavras utilizadas uma Ãºnica vez dividido pelo nÃºmero total de palavras.Tamanho mÃ©dio de sentenÃ§a:MÃ©dia simples do nÃºmero de caracteres por sentenÃ§a.Complexidade de sentenÃ§a:MÃ©dia simples do nÃºmero de frases por sentenÃ§a.Tamanho mÃ©dio de frase:MÃ©dia simples do nÃºmero de caracteres por frase.-InstalaÃ§Ã£o:Use o gerenciador de pacotespippara instalaro text-comparatorAuthorJennifer Figueiredo
Package: text_comparer
Text Comparer=============Uses cosine similarity to give a numerical evaluation of the similarity of twotexts (0 to 1).This code has a companion blog-post here:http://engineering.aweber.com/cosine-similarity/Sample Usage------------```pythonIn [1]: from vectorizer import compare_textsIn [2]: compare_texts('Mary had a little shotgun.', 'Mary loves her shotgun')Out[2]: 0.66666666666666663In [3]: compare_texts('John loves Mary.', 'But Mary has a shotgun.')Out[3]: 0.33333333333333331```The higher score in `2` implies that the first two sentences are more similarthan the second two.  A classic tale of the love-linked-list.
Package: textcompari
A function to get the difference between two files and return a pandas DataFramepip install textcompariTested against Windows 10 / Python 3.11 / AnacondaImportant!The module will be compiled when you import it for the first time. Cython and a C/C++ compiler must be installed!fromrapidfuzzimportfuzzfromtextcompariimportget_file_diff"""A function to get the difference between two files and return a pandas DataFrame.:param afile: A file to compare (str, bytes, tuple, list, np.ndarray):param bfile: Another file to compare (str, bytes, tuple, list, np.ndarray):param window_shifts: The number of shifts for the window (default 5):param min_fuzz_match: The minimum fuzzy match score (default 80):param fuzz_scorer: The fuzzy scorer function (default fuzz.WRatio):param cpus: The number of CPUs to use (default 5):return: A pandas DataFrame containing the difference between the files"""afile=r"C:\Users\hansc\Downloads\difffindertest\test1_1.txt"bfile=r"C:\Users\hansc\Downloads\difffindertest\test1_2.txt"df=get_file_diff(afile,bfile,window_shifts=300,min_fuzz_match=99,fuzz_scorer=fuzz.WRatio,cpus=5)print(df)
Package: text-complexity-computer
Here is TextComplexityComputer.TextComplexityComputer is a state-of-the-art library for assessing complexity of a text in French. Use
it to assess the complexity of a text in French. it is compatible with thelatest version of Scikit-LearnandPython >= 3.8.Getting Started:fromtext_complexity_computerimportTextComplexityComputertcc=TextComplexityComputer()print(tcc.get_metrics_scores("Alibaba et les 40 voleurs."))print(tcc.compute("Alibaba et les 40 voleurs."))InstallationInstall the stable version of TextComplexityComputer:pipinstalltext_complexity_computerInstall the latest development version of TextComplexityComputer:pipinstall-Ugit+https://github.com/GRAAL-Research/tcc.git@devCiteUse the following to cite this package and our article;@article{Primpied2022Quantifying,
	author = {Primpied, Vincent and Beauchemin, David and Khoury, Richard},
	journal = {Proceedings of the Canadian Conference on Artificial Intelligence},
	year = {2022},
	month = {may 27},
	note = {https://caiac.pubpub.org/pub/iaeeogod},
	publisher = {Canadian Artificial Intelligence Association (CAIAC)},
	title = {Quantifying {French} {Document} {Complexity} },
}Contributing to TextComplexityComputerWe welcome user input, whether it is regarding bugs found in the library or feature propositions ! Make sure to have a
look at ourcontributing guidelinesfor more details on this matter.LicenseTextComplexityComputer is LGPLv3 licensed, as found in
theLICENSE file.
Package: textconfig
TextConfigTextConfig is a Class for managing simple text configuration files with
python. By simple, it means that they are simple text files, there are
no sections and all options in the configuration files use a strictOPTION=valueformat, with no spaces betweenOPTION,=andvalue.
The sameOPTIONcan be used more than once, so you can have multiple
values, for example:EXAMPLE=one
EXAMPLE=twoHere's a real-world example of such a file, taken from theslapt-getpackage
manager repository configuration file in Salix:# Working directory for local storage/cache.
WORKINGDIR=/var/slapt-get

# Exclude package names and expressions.
# To exclude pre and beta packages, add this to the exclude:
#   [0-9\_\.\-]{1}pre[0-9\-\.\-]{1}
EXCLUDE=^aaa_elflibs,^aaa_base,^devs,^glibc.*,^kernel-.*,^rootuser-settings,^zzz-settings.*,-i?86-

# The Slackware repositories, including dependency information
SOURCE=http://www.mirrorservice.org/sites/download.salixos.org/x86_64/slackware-14.2/:OFFICIAL
SOURCE=http://www.mirrorservice.org/sites/download.salixos.org/x86_64/slackware-14.2/extra/:OFFICIAL

# The Salix repository
SOURCE=http://www.mirrorservice.org/sites/download.salixos.org/x86_64/14.2/:PREFERRED
# And the Salix extra repository
SOURCE=http://www.mirrorservice.org/sites/download.salixos.org/x86_64/extra-14.2/:OFFICIALYou may call it like this:from textconfig import TextConfig
c = TextConfig('/path/to/configfile')
Package: text-content-generator
TextContentGenerationGenerates Text Content from various sources such as YouTube Videos, Local Audio/Video Files, Webpages and PDF's.ImportsUse thetcg.generate_content()function to generate content from
four sources:PDFSWeb PagesYouTube Video TranscriptsLocal Audio/Video FilesYou can also individual modules if you need only specific functionality.audioaudio_conv - converts files to wav formattranscribe - whisper model to transcribe audio filesyoutubeyt_donwload - Download the yt videosearch_yt - Returns search titles based on search querysearch_and_downloadyoutube_script - Gets transcript/captions content (EN)summarizesummarize_chunks - Summarizes content using BARTwebpageread_pages - Scrapes a pagepdfpdf_read - Reads pdfs
Package: textconv
textconvA bulletproof way of converting a string to a byte string or a text string.APIs are designed to encode or decode raise-free.
In most cases, input object will be either a text string (strtype) or
a byte string (bytestype), and default encoding is'utf-8'.Usage:fromtextconv.textimportto_text,to_bytes# To text stringtext_string=to_text(b'caf\xc3\xa9')# To byte stringbyte_string=to_bytes(u'caf\xe9',encoding="latin-1")# A raise-free version of `str(obj)`# if a non-string object is  providedtext_string=to_text(obj)
Package: textcrafts
TextGraphCraftsPython-based summary, keyphrase and relation extractor from text documents using dependency graphs.HOME:https://github.com/ptarau/TextGraphCraftsProject Description** The system uses dependency links for building Text Graphs, that with help of a centrality algorithm likePageRank, extract relevant keyphrases, summaries and relations from text documents.  Developed withPython 3, on OS X, but portable to Linux.**Dependencies:python 3.7 or newer, pip3, java 9.x or newer. Also, having git installed is recommended for easy updatespip3 install nltkalso, run in python3 something likeimport nltk
nltk.download('wordnet')
nltk.download('words')
nltk.download('stopwords')or, if that fails on a Mac, use runpython3 down.pyto collect the desired nltk resource files.pip3 install networkxpip3 install requestspip3 install graphviz, also ensure .gv files can be viewedpip3 install stanfordnlpparserNote thatstanfordnlprequires torch binaries which are easier to instal with ````anaconda```.Tested with the above on a Mac, with macOS Mojave and Catalina and on Ubuntu Linux 18.x.Running it:in a shell window, runstart_server.shin another shell window, start withpython3 -i tests.pyand then interactively, at the ">>>" prompt, try>>> test1()
>>> test2()
>>> ...
>>> test9()
>>> test12()
>>> test0()see how to activate other outputs in filedeepRank.pytext file inputs (including the US Constitution const.txt) are in the folderexamples/Handling PDF documentsThe easiest way to do this is to installpdftotext, which is part ofPoppler tools.If pdftotext is installed, you can place a file liketextrank.pdfalready in subdirectory pdfs/ and try something similar to:Change setting in file params.py to use the system with
other global parameter settings.Alternative NLP toolkitOptionally, you can activate the alternative Stanford CoreNLP toolkit as follows:installStanford CoreNLPand unzip in a derictory of your choice (ag., the local directory)edit if neededstart_parser.shwith the location of the parser directoryoverride theparamsclass and setcorenlp=TrueNote however that the Stanford CoreNLP is GPL-licensed, which can place restrictions on proprietary software activating this option.
Package: textcut
SummaryText Cut is meant to wrap textnaturally, preferring to cut the sentence where this is the least
disturbing for the reader.Installpython-mpipinstalltextcutUsagefromtextcutimportTextCutprint(TextCut().fill(my_text))MethodsThetextcutpackage offers two main methods:wrapcuts a long text into multiple lines,
andfilldoes the same but join the resulting list with new line character for easy printing.OptionsSeveral options are available to configure the wrapping:language: for now onlyfr(French), the rules to wrap text depends on the language. Althoughfrworks quite well with English too.width: the maximum width of the linestolerance: whether it's ok to have shorter lines that other (high tolerance >0.5) or not (low tolerance, <0.1)trim: whether to trim the lines to remove extra spaceslen_func: a function to determine the length of a string, defaults tolennormalise: tries to minimise the length differences between lines (not optimised)ExampleTextCut(language='fr',width=80,tolerance=0.4,trim=True,len_func=len).fill(text)Output:If I were asked to answer the following question: What is slavery?
and I should answer in one word, It is murder!,
my meaning would be understood at once.
No extended argument would be required to show that the power to remove a man's
mind, will, and personality, is the power of life and death,
and that it makes a man a slave. It is murder. Why, then,
to this other question: What is property? may I not likewise answer,
It is robbery!, without the certainty of being misunderstood;
the second proposition being no other than a transformation of the first?
Package: textda
This is Data Augmentation for Chinese text for Python3Usageyou have two func for Chinese text Data AugmentationInstall textdapip install:pipinstalltextdayou can expansion data usedata_expansionfromtextda.data_expansionimport*print(data_expansion('生活里的惬意，无需等到春暖花开'))output:['生活里面的惬意，无需等到春暖花开','生活里的等到春暖花开','生活里无需惬意，的等到春暖花开','生活里的惬意，无需等到春暖花开','生活里的惬意，并不需要等到春暖花开','生活无需的惬意，里等到春暖花开','生活里的惬意，等到无需春暖花开']param explain：:param sentence: input sentence text
:param alpha_sr: Replace synonym control param. bigger means more words are Replace
:param alpha_ri: Random insert. bigger means more words are Insert
:param alpha_rs: Random swap. bigger means more words are swap
:param p_rd: Random delete. bigger means more words are deleted
:param num_aug: How many times do you repeat each methodyou can use parameters alpha_sr, alpha_ri, alpha_rs, p_rd, num_aug can control ouput.if you set alpha_ri and alpha_rs is 0 that means uselinear classifierfor it, and insensitive to word locationlike this:fromtextda.data_expansionimport*print(data_expansion('生活里的惬意，无需等到春暖花开',alpha_ri=0,alpha_rs=0))output:['生活里的惬意，无需等到春暖花开','，无需春暖花开','生活里面的惬意，无需等到春暖花开','生活里的惬意，需等到春暖花开']you can usetranslate_batchlike this:fromtextda.youdao_translateimport*dir='./data'translate_batch(os.path.join(dir,'insurance_train'),batch_num=30)# translate results:  chinese->english and english -> chinese

颜色碰掉了一个角不延迟,但事情或他们不赠送,或发送,眉笔打开已经破碎,磨山楂,也不打破一只手,轻轻刷掉,持久性不长,
这个用户没有填写评价内容
颜色非常不喜欢它
不说话,缓慢的新领域
不太容易染好骑吗
不是很好我喜欢!
没有颜色的眼影
应该有大礼物盒眼影,礼物不礼物盒,没有一起破碎粉碎好的眼影不买礼物清洁剂脏就像商品是压力
没有生产日期,我不知道是否真实,总是觉得有点奇怪
是一个小飞粉吗
但是一些混合的颜色
有几次,现在这个东西,笔是空的
眼影有点小,少一点。
不好的颜色,粉红色
明星不想买,坏了,不容易,不要在乎太多!
一开始我已经联系快递,快递一直拖,说他将返回将联系快递服务
画不是,是不好的
物理和照片有很大的区别
不要把眼影刷不是很方便
感觉好干,颜色更暗
打破了在运输途中,有点太脆弱…
盒子有点坏了,还没有发送。param explain：:param file_path: src file path
:param batch_num: default 30
:param reWrite: default True. means you can rewrite file , False means you can append data after this file.
:param suffix: new file suffixReference:https://github.com/jasonwei20/eda_nlpCode for the ICLR 2019 Workshop paper: Easy data augmentation techniques for boosting performance on text classification tasks.https://arxiv.org/abs/1901.11196LicenseMIT
Package: text-denoising
Masking implementation for Unifying Language Learning Paradigms (UL2)Want to get better model with limited budgets? You are in the right placeR-Denoiser (μ=3,r=0.15,n)∪ (μ=8,r=0.15,n)The regular denoising is the standard span corruption introduced in Raffel et al. (2019) that uses a range of 2 to 5 tokens as the span length, which masks about 15% of input tokensS-Denoiser (μ=L/4,r=0.25,1)A specific case of denoising where we observe a strict sequential order when framing the inputs-to-targets task, i.e., prefix language modelingX-Denoiser (μ = 3,r = 0.5,n)∪(μ = 8,r = 0.5,n)∪(μ = 64,r =0.15,n)∪ (μ=64,r=0.5,n)An extreme version of denoising where the model must recover a large part of the input, given a small to moderate part of it. This simulates a situation where a model needs to generate long target from a memory with relatively limited information. To do so, we opt to include examples with aggressive denoising where approximately 50% of the input sequence is masked2022 papers : Transcending Scaling Laws with 0.1% Extra Computewe show an approximately 2x computational savings rateRegular denoising whereby the noise is sampled as spans, replaced with sentinel tokens. This is also the standard span corruption task used in Raffel et al. (2019). Spans are typically uniformly sampled with a mean of 3 and a corruption rate of 15%.Extreme denoising whereby the noise is increased to relatively ‘extreme‘ amounts in either a huge percentage of the original text or being very long in nature. Spans are typically uniformly sampled with a mean length of 32 OR a corruption rate of up to 50%.Sequential denoising whereby the noise is always sampled from the start of the text to a randomly sampled point in the text. This is also known as the PrefixLM objective (not to be confused with the architecture).This repo will just aim for accompolish this task instead, UL2 is way too complicated for my likings50% PrefixLM, 25% Long (extreme) span corruption, and 25% regular span corruption to be quite simple and efficientExperimentsRun a mT5 encoder pretraining on 3090 on pythia json.zst filespython pretrain_example.pytraining loss was stable and no weird spikesReferencesCore PapersTranscending Scaling Laws with 0.1% Extra ComputeUnifying Language Learning ParadigmsImplements of t5 noise masking in huggingface transformers or python codeOSLO: very underrated, some tidy and documentation, this will be a very useful toolt5_pretraining.pyHeavily inspired from this sectionAmazon science : label aware pretrain in pythonFairseq : span_mask_tokens_dataset.py
Package: text-detector
text-detector (algorithm accuracy work in progress)Locating texts in images using machine vision algorithmsThis project aims to use only image processing techniques to locate text regions in the image. More detailed information about the approach is given this linkhttps://muthu.co/extracting-text-regions-from-an-image-using-geometric-properties/Installationpip install -r requirements.txtUsageimport matplotlib.patches as mpatches
import matplotlib.pyplot as plt
from skimage.io import imread

from text_detector import detect_text_regions

image_sample = 'sample_images/image7.jpg'
image = imread(image_sample)
box_groups = detect_text_regions(image)

fig, ax = plt.subplots()
for box in box_groups:
    minr = box[0]
    minc = box[1]
    maxr = box[2]
    maxc = box[3]
    rect = mpatches.Rectangle((minc, minr), maxc - minc, maxr - minr,
                              fill=False, edgecolor='red', linewidth=2)
    ax.add_patch(rect)

ax.add_patch(rect)
ax.imshow(image)
ax.set_axis_off()
plt.tight_layout()
plt.show()Sample ResultsOriginalDetected Text RegionsNotebookAll my experiments are in this notebook also part of the project where I make changes to the algorithm and them move it to the detector file.Text Segmentation in Image.ipynbAccuracyThe accuracy depends a lot on the threshold parameters in theutils.thresholds.pyfile which may need tweaking for different kinds of datasets.ReferencesB. Epshtein, E. Ofek and Y. Wexler, "Detecting text in natural scenes with stroke width transform," 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Francisco, CA, 2010, pp. 2963-2970.Tran, Tuan Anh Pham et al. “Separation of Text and Non-text in Document Layout Analysis using a Recursive Filter.” TIIS 9 (2015): 4072-4091.Chen, Huizhong, et al. "Robust Text Detection in Natural Images with Edge-Enhanced Maximally Stable Extremal Regions." Image Processing (ICIP), 2011 18th IEEE International Conference on. IEEE, 2011.
Package: text-det-metric
Text Detect MetricThis library is used to calculate the three metricPrecision,RecallandH-meanto evaluate the effect of text detection algorithms. It is used in conjunction withModelscope-Text Detection Test Set.Indicator calculation code reference:PaddleOCRandDBEvaluate on the custom dataset.Here we use the evaluation code ofch_ppocr_v3_deton the text detection test setliekkas/text_det_test_dataset, and you can use the same analogy.UsageInstall packages.pipinstallmodelscope==1.5.2
pipinstalltext_det_metricRunget_pred_txt.pyto getpred.txtClick to expandfrompathlibimportPathimportcv2importyamlfrommodelscope.msdatasetsimportMsDatasetfromtqdmimporttqdmfromdet_demos.ch_ppocr_v3_detimportTextDetectorroot_dir=Path(__file__).resolve().parentdefread_yaml(yaml_path):withopen(yaml_path,"rb")asf:data=yaml.load(f,Loader=yaml.Loader)returndatatest_data=MsDataset.load("text_det_test_dataset",namespace="liekkas",subset_name="default",split="test",)config_path=root_dir/'det_demos'/'ch_ppocr_v3_det'/'config.yaml'config=read_yaml(str(config_path))# Configure the onnx model path.config['model_path']=str(root_dir/'det_demos'/config['model_path'])text_detector=TextDetector(config)content=[]forone_dataintqdm(test_data):img_path=one_data.get("image:FILE")img=cv2.imread(str(img_path))dt_boxes,elapse=text_detector(img)content.append(f"{img_path}\t{dt_boxes.tolist()}\t{elapse}")withopen("pred.txt","w",encoding="utf-8")asf:forvincontent:f.write(f"{v}\n")Runcompute_metric.pyto get the metrics on the datasetfromtext_det_metricimportDetectionIoUEvaluatormetric=DetectionIoUEvaluator()# pred_pathpred_path="pred.txt"metric=metric(pred_path)print(metric)Output{'precision':0.6958333333333333,'recall':0.8608247422680413,'hmean':0.7695852534562212,'avg_elapse':2.0107483345529307}See details forTextDetMetric.
Package: text-diff
Text DiffPython diffline wrapper.Quick Startfromtext_diffimporttext_differencestext_1=['Hello','World','How are you ?',]text_2=['Hello','World!','How are u ?',]diff=text_differences(text_1,text_2)forlineindiff.diff_lines:print(type(line))InstallationStable Release:pip install text_diffDevelopment Head:pip install git+https://github.com/envinorma/text_diff.gitDocumentationFor full package documentation please visitenvinorma.github.io/text_diff.DevelopmentSeeCONTRIBUTING.mdfor information related to developing the code.MIT license
Package: textdiversity
TextDiversityInstallationpip install -r requirements.txtSome of the text diversity measures rely on software that must be installed separately. For your particular OS, follow the instructions below:WindowsPhoneme DiversityPhonemizerVisit:https://github.com/espeak-ng/espeak-ng/releasesInstall eitherespeak-ng-X64.msiorespeak-ng-X86.msi(This project used Release 1.51)Add Phonemizer environement variable ([reference])Navigate: Windows Key > Edit System Environment Variables > Environment VariablesAdd New Variable:Variable Name:PHONEMIZER_ESPEAK_LIBRARYVariable Value:C:\Program Files\eSpeak NG\libespeak-ng.dllSyntactical Diversity (non-core functionality)Dependency Parse Tree VisualizationUNKOWN SETUP STEPSLinuxPhoneme DiversityPhonemizersudo apt-get install espeak-ngSyntactical DiversityDependency Parse Tree Visualization (non-core functionality)sudo apt-get install python3-dev graphviz libgraphviz-dev pkg-config
Package: textdungeon
TODO: rewrite slogan and remove repo name hyphen or rename game entirely.textdungeon: suffer alone in your terminal!AboutThis is a game created by Matthew Bender that allows you to traverse a spoopy dungeon. yay!! Rewrite me please... Noah was here.InstallationPython Pipinstall from pypipipinstalltextdungeonthen simply run withtextdungeonWindowsDownload text-dungeon.exe fromlatest