Saravanan666/pypi-org huggingface dataset:
Package: dedoc-utils
dedoc-utilsThis library contains useful utilities for automatic document images processing:Preprocessingbinarizationskew correctionText detectionLine segmentationText recognitionInstallationThe library requires Tesseract OCR to be installed.
To install the library use the following command:pipinstalldedoc-utilsIt's supposed that you already havetorchandtorchvisioninstalled.
If not you can use the following command for installation:pipinstall"dedoc-utils[torch]"If you cloned the repository, you can install the dependencies viapip:pipinstall.To installtorchpackages use:pipinstall."[torch]"Basic usageUsing preprocessorsfromdedocutils.preprocessingimportAdaptiveBinarizer,SkewCorrectorimportcv2importmatplotlib.pyplotaspltbinarizer=AdaptiveBinarizer()skew_corrector=SkewCorrector()image=cv2.imread("examples/before_preprocessing.jpg")binarized_image,_=binarizer.preprocess(image)preprocessed_image,_=skew_corrector.preprocess(binarized_image)fig=plt.figure(figsize=(10,7))rows,columns=1,3fig.add_subplot(rows,columns,1)plt.imshow(image)plt.axis('off')plt.title("Before preprocessing")fig.add_subplot(rows,columns,2)plt.imshow(binarized_image)plt.axis('off')plt.title("After binarization")fig.add_subplot(rows,columns,3)plt.imshow(preprocessed_image)plt.axis('off')plt.title("After preprocessing")Using text detectorsfromdedocutils.text_detectionimportDoctrTextDetectortext_detector=DoctrTextDetector()bboxes=text_detector.detect(preprocessed_image)forbboxinbboxes[:5]:print(bbox)BBox(x_top_left=2415, y_top_left=3730, width=202, height=97)BBox(x_top_left=790, y_top_left=3613, width=383, height=105)BBox(x_top_left=1690, y_top_left=3488, width=407, height=104)BBox(x_top_left=2171, y_top_left=3488, width=377, height=92)BBox(x_top_left=885, y_top_left=3505, width=27, height=50)Using text recognizersfromdedocutils.text_recognitionimportTesseractTextRecognizertext_recognizer=TesseractTextRecognizer()forbboxinbboxes[:10]:word_image=preprocessed_image[bbox.y_top_left:bbox.y_bottom_right,bbox.x_top_left:bbox.x_bottom_right]text=text_recognizer.recognize(word_image,parameters=dict(language="eng"))print(text)Fie-afjefjores.coluntur,dicunturdelubroeodemdii
inpluresUsing line segmentersIn the previous example, the order of the recognized words isn't the same
as the order of the words in the document.
It happens because of undetermined work of the text detector.
In this case, one may use line segmenter to sort bboxes from the text detector.fromdedocutils.line_segmentationimportClusteringLineSegmenterline_segmenter=ClusteringLineSegmenter()sorted_bboxes=line_segmenter.segment(bboxes)forbboxinsorted_bboxes[1]:word_image=preprocessed_image[bbox.y_top_left:bbox.y_bottom_right,bbox.x_top_left:bbox.x_bottom_right]text=text_recognizer.recognize(word_image,parameters=dict(language="eng"))print(text)gentes,fimul.obibantmunerafummafacra,
Package: dedomenon
Dedomenon: A Python Library for DatasetsDocumentation:https://dedomenom.entelecheia.aiGitHub:https://github.com/entelecheia/dedomenonPyPI:https://pypi.org/project/dedomenonDedomenon is a versatile and powerful Python library designed for managing, processing, and analyzing datasets, catering to the diverse needs of data scientists, researchers, and developers across various domains. The name Dedomenon is inspired by the Greek word δεδομένων (data), reflecting the library's core focus on facilitating efficient and effective data handling.FeaturesDedomenon offers a comprehensive set of features, making it an ideal choice for users with varying levels of expertise in data science and analytics:User-friendly API: Dedomenon provides an intuitive and easy-to-use API, enabling users to quickly load, manipulate, and analyze datasets with minimal lines of code.Support for various data formats: The library supports a wide range of data formats, including CSV, JSON, Excel, Parquet, and SQL, allowing users to work with datasets in their preferred format.Efficient data processing: Dedomenon utilizes advanced algorithms and data structures to ensure high-performance data processing, even on large-scale datasets.Data cleaning and preprocessing: The library offers a variety of tools for data cleaning, preprocessing, and transformation, helping users to prepare their datasets for further analysis and modeling.InstallationYou can install Dedomenon using pip:pipinstalldedomenonGetting StartedTo begin working with Dedomenon, visit theofficial documentationand theGitHub repositoryfor examples, tutorials, and additional information.ChangelogSee theCHANGELOGfor more information.ContributingContributions are welcome! Please see thecontributing guidelinesfor more information.LicenseThis project is released under theMIT License.
Package: dedoop
dedoopIndigital preservationwork you sometimes may find yourself accepting a disk
or random assortment of files, and want to examine all of them looking for
duplicates and copy them to a new location in a uniform way, while preserving
the original paths as metadata to help you process the data. Ok, maybe this is a
bit of niche use case, but this is whatdedoopwas created for.dedoopwill recursively read a source directory of files and write them out to
a new target directory or bucket in the cloud using the files's SHA256 checksum
as the filename. If a given file occurs more than once in the source
directory it will only be written once to the target location. File metadata
such as the media type and original file name will be persisted in a JSON file
that is output at the end of the process. In the case of writing to the cloud,
object metadata will be used to store this information.InstallInstall Python 3 and:% pip3 install dedoopUsageAdd to StorageTo add a directory of data to the storage location you can:% dedoop add path/to/source path/to/targetSo for example if the source directory looks like this:source
├── a.jpg
├── a.png
├── b.jpg
└── c
    ├── a.jpg
    └── b.jpgThe resulting target could look like this (assuming the files of the same name
had the same contents that hashed to these values):target
├── 1e89b90b5973baad2e6c3294ffe648ff53ab0b9d75188e9fbb8b38deb9ba3341.png
├── 45d257c93e59ec35187c6a34c8e62e72c3e9cfbb548984d6f6e8deb84bac41f4.jpg
└── b6df8058fa818acfd91759edffa27e473f2308d5a6fca1e07a79189b95879953.jpgAdd to the CloudYou can also write files to any cloud storage provider that issupportedbylibcloud,
such as Amazon S3, Google Cloud Storage, etc.Limit by File ExtensionIf you like you can limit the types of files that are added by using the--extensionscommand line option and giving it a comma separated list of file
extensions to include. All non-matching files (case insensitive) will be
ignored.% dedoop add --extensions jpg,png path/to/source path/to/targetList Cloud FilesIts easy to list files on the file system. But its more difficult to see what's
in the cloud--especially with the  metadata dedoop has attached to each object.
Thelistcommand will do that for you.% dedoop ls s3://my-storage-location/LoggingIf you use--verboseyou will see log messages on the console about what is
happening. You can optionally send these messages to a log file of your choosing
using the--logoption.
Package: dedscumulus
IntroductionTODO: Give a short introduction of your project. Let this section explain the objectives or the motivation behind this project.Getting StartedTODO: Guide users through getting your code up and running on their own system. In this section you can talk about:Installation processSoftware dependenciesLatest releasesAPI referencesBuild and TestTODO: Describe and show how to build your code and run the tests.ContributeTODO: Explain how other users and developers can contribute to make your code better.If you want to learn more about creating good readme files then refer the followingguidelines. You can also seek inspiration from the below readme files:ASP.NET CoreVisual Studio CodeChakra Core
Package: deduck
deduckHi! explain your problem to me... quack! quack!Authors@ArticOffInstallationInstall info with pip# Linux/macOSpython3-mpipinstall-Udeduck# Windowspy-3-mpipinstall-UdeduckUsage/ExamplesOpen a terminalWrite thispython-mduckThat's all !FeedbackIf you have any feedback, please reach out to us onour discordLinksOfficial discord serverCodeSec CommunitySource
Package: dedupe-fh
dedupe is a library that uses machine learning to perform de-duplication and entity resolution quickly on structured data. dedupe is the open source engine fordedupe.iodedupewill help you:remove duplicate entriesfrom a spreadsheet of names and addresseslink a listwith customer information to another with order history, even without unique customer id’stake a database of campaign contributions andfigure out which ones were made by the same person, even if the names were entered slightly differently for each recorddedupe takes in human training data and comes up with the best rules for your dataset to quickly and automatically find similar records, even with very large databases.
Package: dedupe-fork-eccovia
dedupe is a library that uses machine learning to perform de-duplication and entity resolution quickly on structured data. dedupe is the open source engine fordedupe.iodedupewill help you:remove duplicate entriesfrom a spreadsheet of names and addresseslink a listwith customer information to another with order history, even without unique customer id’stake a database of campaign contributions andfigure out which ones were made by the same person, even if the names were entered slightly differently for each recorddedupe takes in human training data and comes up with the best rules for your dataset to quickly and automatically find similar records, even with very large databases.
Package: dedupe-FuzzyWuzzy
dedupe-FuzzyWuzzyDeduplication of a data set using rapidfuzz library which is just the same as FuzzyWuzzy but is a lot faster.Installationpip install dedupe-FuzzyWuzzyBasic UsageIt is very simple to use, you just have to import the library and pass the dataframe in dedupeFuzzy along with a list of columns which will be used for deduplication and thershold for how strict you want the criteria to be .
A higher threshold will be strict in matching and will give you less matches whereas a lower threshold will give you more matches .Deduplicationimport pandas as pd
import dedupe_FuzzyWuzzy
from dedupe_FuzzyWuzzy import deduplication

#load dataframe
df = pd.read_csv('messy.csv')

#initiate deduplication
df_1 = deduplication.deduplication(df,['Site name','Address'],threshold=90)

#send output to csv
df_1.to_csv('dedupeOutput.csv')CreditsThis would have not been possible without the rapidfuzz package whose author is @maxbachmann, so kudos to you !
Package: dedupe-hcluster
This library provides Python functions for hierarchical clustering. Its features
includegenerating hierarchical clusters from distance matricescomputing distance matrices from observation vectorscomputing statistics on clusterscutting linkages to generate flat clustersand visualizing clusters with dendrograms.The interface is very similar to MATLAB’s Statistics Toolbox API to make code
easier to port from MATLAB to Python/Numpy. The core implementation of this
library is in C for efficiency.
Package: deduper
OneFL Deduper------------OneFlorda Entity Resolution Software (aka Deduper)<https://github.com/ufbmi/onefl-deduper> is designed to generate hashes of thespecific patient data elements.Multiple generated hashes are then matched against the registry of "known"hashes and associated with a Unique Universal Identifier (aka OneFlorida UUID).The UUID is used to de-duplicate medical records such asdemographics, procedures, diagnoses, vitals, lab results submittedfrom different hospital networks.
Package: dedupe-variable-ilcs
dedupe-variable-ilcsDedupe variable for Illinois Compiled Statute (ILCS) codes.Part of theDedupe.iocloud service and open source toolset for de-duplicating and finding fuzzy matches in your data.InstallationInstall withpip:pip install git+https://github.com/dedupeio/dedupe-variable-ilcs.gitDevelopmentLocal development requires an installation of Python.Install development requirements:pip install -e .[tests]Train theilcs_parsermodel:parserator train training/labeled.xml ilcs_parserThe variable should now be ready to use.Running testsRun the tests:pytestAll tests should pass.
Package: dedup_hash
.. image:: https://travis-ci.org/mvdbeek/dedup_hash.svg?branch=master:target: https://travis-ci.org/mvdbeek/dedup_hashdedup_hash----------------------------This is a commandline utility to remove exact duplicate readsfrom paired-end fastq files. Reads are assumed to be in 2 separatefiles. Read sequences are then concatenated and a short hash is calculated onthe concatenated sequence. If the hash has been previsouly seen the read willbe dropped from the output file.  This means that reads that have the samestart and end coordinate, but differ in lengths will not be removed (but thosewill be "flattened" to at most 1 occurence).This algorithm is very simple and fast, and saves memory as compared toreading the whole fastq file into memory, such as fastuniq does.Installation------------depdup_city relies on the cityhash python package,which supports python-2.7 exclusively.``pip install dedup_hash``History-------.. to_doc---------------------0.1.1 (2016-11-23)---------------------* Make python2/3 compatible* Use smhasher as default hasher and add options for cityhash and hashxx* Testing enhancements* Initial version---------------------0.1.0 (2016-11-16)---------------------* Initial version
Package: dedup-img
dedup-img描述dedup-img是一个基于imagededup库做封装的图片去重库，使用本地web做UI展示，所以该库依赖于:imagededup: 图片去重flask: 生成本地web，用于展示和操作图片去重后的分类展示pillow: 生成缩略图安装pip install dedup-img使用图片去重命令dedup -P -d e:/Pictures选项解析-P 图片去重算法选项使用说明-P感知哈希（PHash）-A平均哈希（AHash）-D差异哈希（DHash）-W小波哈希（WHash）-d 后接需要去重的图片的所在文件夹执行去重命令后会在图片所在文件夹里生成若干文件浏览去重后的图片数据展示flask-web e:/Pictures
Package: deduplicate
deduplicatevpypeplug-in to remove overlapping lines in SVG files.UsageUsage: vpype deduplicate [OPTIONS]

  Remove duplicate lines.

Options:
  -t, --tolerance LENGTH  Max distance between points to consider them equal
                          (default: 0.01mm)
  -p, --progress-bar      (flag) Display a progress bar
  -l, --layer LAYERS      Target layer(s) (defaul: 'all')
  -k, --keep-duplicates   (flag) Keep removed duplicates in a separate layer
  --help                  Show this message and exit.ExamplesBasic usage :vpype read file.svg deduplicate write output.svgYou can keep removed lines in a separate layer with-kflag:vpype read squares.svg deduplicate -k write squares_dedup.svgThe-toption controls the tolerance, which is the maximum distance 2 points are considered equal.Here is whatvpype read tangent_circles.svg deduplicate -k -t 0.2mm showoutputs with different tolerances:InstallationSee theinstallation instructionsfor information on how
to installvpype.ExistingvpypeinstallationIfvpypewas installed using pipx, use the following command:$pipxinjectvpypededuplicateIfvpypewas installed using pip in a virtual environment, activate the virtual environment and use the following command:$pipinstalldeduplicateCheck that your install is successful:$ vpype --help
Usage: vpype [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options:
  -v, --verbose
  -I, --include PATH  Load commands from a command file.
  --help              Show this message and exit.

Commands:
[...]
  Plugins:
    deduplicate
[...]Stand-alone installationUse this method if you need to edit this project. First, clone the project:$gitclonehttps://github.com/LoicGoulefert/deduplicate.git
$cddeduplicateCreate a virtual environment:$python3-mvenvvenv
$sourcevenv/bin/activate
$pipinstall--upgradepipInstalldeduplicateand its dependencies (includingvpype):$pipinstall-e.Check that your install is successful:$ vpype --help
Usage: vpype [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...

Options:
  -v, --verbose
  -I, --include PATH  Load commands from a command file.
  --help              Show this message and exit.

Commands:
[...]
  Plugins:
    deduplicate
[...]LicenseSee theLICENSEfile for details.
Package: deduplicated
Check duplicated files.Exemple of use# Update directory cache$deduplicatedupdate/path/for/check# List duplicated files in directory cache$deduplicatedduplicated/path/for/check# Update and list duplicated files$deduplicatedcheck/path/for/check# Check if file in directory cache$deduplicatedindirmyfile/path/for/check# Start web version, connect http://127.0.0.1:5050$deduplicated-web
Package: deduplication
deduplicationRemove duplicate documents via popular algorithms such as SimHash, SpotSig, Shingling, etc.InstallRun following commands:# install current library
pip install deduplication

# install required pretrained NLP models 
python -m spacy download xx_ent_wiki_sm
python -m spacy download en_core_web_smExampleSimHashfromdeduplicationimportsimhashhashvalue1=simhash('this is text')hashvalue2=simhash('this is another text',n_block=4)L-SimHashfromdeduplicationimportlsimhashhashvalue=lsimhash('this is very long article texts. maybe with a lot of sentences.')CitationSimHashSadowski C, Levin G. 
Simhash: Hash-based similarity detection[J]. 
Technical report, Google, 2007.
Package: deduplicationdict
DeDuplicationDictA dictionary that de-duplicates values.A dictionary-like class that deduplicates values by storing them in a separate dictionary and replacing
them with their corresponding hash values. This class is particularly useful for large dictionaries with
repetitive entries, as it can save memory by storing values only once and substituting recurring values
with their hash representations.This class supports nested structures by automatically converting nested dictionaries intoDeDuplicationDictinstances. It also provides various conversion methods to convert between regular
dictionaries andDeDuplicationDictinstances.InstallationpipinstalldeduplicationdictUsagefromdeduplicationdictimportDeDuplicationDict# Create a new DeDuplicationDict instancededup_dict=DeDuplicationDict.from_dict({'a':[5,6,7],'b':2,'c':[5,6,7]})# ordedup_dict=DeDuplicationDict(**{'a':[5,6,7],'b':2,'c':[5,6,7]})# Add a new duplicate key-value pairdedup_dict['d']=[1,2,3]dedup_dict['e']=[1,2,3]# Print the dictionaryprint(f"dedup_dict.to_dict():{dedup_dict.to_dict()}")# output: {'a': [5, 6, 7], 'b': 2, 'c': [5, 6, 7], 'd': [1, 2, 3], 'e': [1, 2, 3]}# Print the deduplicated dictionary internalprint(f"dedup_dict.key_dict:{dedup_dict.key_dict}")# output: {'a': '7511debb', 'b': '7c7ad8f0', 'c': '7511debb', 'd': 'f9343d7d', 'e': 'f9343d7d'}print(f"dedup_dict.value_dict:{dedup_dict.value_dict}")# output: {'7511debb': [5, 6, 7], '7c7ad8f0': 2, 'f9343d7d': [1, 2, 3]}# Print the deduplicated dictionaryprint(f"to_json_save_dict:{dedup_dict.to_json_save_dict()}")# output: {'key_dict': {'a': '7511debb', 'b': '7c7ad8f0', 'c': '7511debb', 'd': 'f9343d7d', 'e': 'f9343d7d'}, 'value_dict': {'7511debb': [5, 6, 7], '7c7ad8f0': 2, 'f9343d7d': [1, 2, 3]}}assertdedup_dict["a"]==[5,6,7]assertdedup_dict["b"]==2assertdedup_dict["c"]==[5,6,7]assertdedup_dict["d"]==[1,2,3]assertdedup_dict["e"]==[1,2,3]assertDeDuplicationDict.from_json_save_dict(dedup_dict.to_json_save_dict()).to_dict()==dedup_dict.to_dict()Usage withSqliteDict:SqliteDeDuplicationDict.pyResults from TestingMethodJSON Memory (MB)In-Memory (MB)dict14.089 MB27.542 MBDeDuplicationDict1.7906 MB3.806 MBMemory Saving7.868x7.235xDocumentationThe documentation for this project is hosted onRead the Docs.LicenseThis project is licensed under the terms of theMozilla Public License 2.0.
Package: dedup-me
dedup_mededup_me is a simple library for concurrent in-flight deduplication of functions.
This can be useful for e.g. API calls or DB access. Instead of querying the same data multiple times,
this library allows you to query once and share the result.Note: This library does not cache results. After the result is returned, new consumers will call the function(API, DB, ...) again.Installationpipinstalldedup-meUsageAsyncIOimportasyncioimportrandomfromdedup_meimportasync_dedup# @async_dedup('static-key')  # if the arguments don't matter, you can use a static key@async_dedup(key=lambdax:f'dynamic-key-{x}')# or pass in a key function that accepts all argumentsasyncdefexpensive_function(x:int)->int:print('expensive function called')awaitasyncio.sleep(x)returnrandom.randint(0,10)asyncdefmain()->None:# call the function 10 times# this should only print 'expensive function called' onceresults=awaitasyncio.gather(*(expensive_function(1)for_inrange(10)))print(results)# all results should be the sameresults_list=list(results)assertresults_list==[results_list[0]]*10Alternatively, without the decorator:importasyncioimportrandomfromdedup_meimportAsyncDedupasyncdefexpensive_function()->int:print('expensive function called')awaitasyncio.sleep(1)returnrandom.randint(0,10)asyncdefmain()->None:dedup=AsyncDedup()awaitasyncio.gather(*(# functions are grouped by the key, choose something that represents the function and its arguments# the second argument must be a function without arguments that returns an awaitablededup.run('some-key',lambda:expensive_function())for_inrange(10)))ThreadsFor threading just use thethreading_dedupdecorator orThreadingDedup.fromtimeimportsleepfromdedup_meimportthreading_dedup,ThreadingDedup@threading_dedup('key')defexpensive_function()->None:sleep(1)expensive_function()# ordedup=ThreadingDedup()dedup.run('key',lambda:sleep(1))Forcing a new executionYou can enforce a new execution by passingforce_new = Trueto therunfunction.
When using the decorator, you can add a callable that will receive all arguments and returns a boolean.
Package: dedust
DeDust SDK for PythonAnalogue of DeDust SDK for Python.DescriptionYou can swap tokens, deposit liquidity and many more.Getting StartedDependencieshttpxpytoniqInstallingpip install dedustUsingExample of buying $SCALE using SDK.fromdedustimportAsset,Factory,PoolType,SwapParams,VaultNativefrompytoniqimportWalletV4R2,LiteBalancerimportasyncioimporttimemnemonics=["your","mnemonics","here"]asyncdefmain():provider=LiteBalancer.from_mainnet_config(1)awaitprovider.start_up()wallet=awaitWalletV4R2.from_mnemonic(provider=provider,mnemonics=mnemonics)SCALE_ADDRESS="EQBlqsm144Dq6SjbPI4jjZvA1hqTIP3CvHovbIfW_t-SCALE"TON=Asset.native()SCALE=Asset.jetton(SCALE_ADDRESS)pool=awaitFactory.get_pool(pool_type=PoolType.VOLATILE,assets=[TON,SCALE],provider=provider)swap_params=SwapParams(deadline=int(time.time()+60*5),recipient_address=wallet.address)swap_amount=int(float(input("Enter swap amount: "))*1e9)swap=VaultNative.create_swap_payload(amount=swap_amount,pool_address=pool.address,swap_params=swap_params)swap_amount=int(swap_amount+(0.25*1e9))# 0.25 = gas_valueawaitwallet.transfer(destination="EQDa4VOnTYlLvDJ0gZjNYm5PXfSmmtL6Vs6A_CZEtXCNICq_",# native vaultamount=swap_amount,body=swap)asyncio.run(main())Authors@shibdev@VladPavlyVersion History1.1.2Bug fix1.1.1Bug fix1.1.0Change tonsdk to pytoniq1.0.5Changes1.0.4Bug fix1.0.3Fixes1.0.2Fixes and changes1.0.1Small fix1.0.0Remake to DeDust SDK analogue0.0.5License change0.0.4Dependecies fix0.0.3Examples fix0.0.2LP token address getting0.0.1Initial BetaLicenseThis project is licensed under the MIT License - see the LICENSE.md file for detailsDonateIf you like the library, I will be glad to accept donations.TON: EQCgphx8rTI0PukwmgpVqiPgqguTujhQscg2h7jgc4U0t347Acknowledgmentsdedust-sdkdedust-docs