Saravanan666/pypi-org huggingface dataset:
Package: dupan
度盘令目的用程序做百度盘 Web 界面可以做的所有（Almost）事情。模块化。读。列表、查看写。复制、移动、删除、分享。方式抓取 iPhone 客户端的 HTTP 请求，录下 HAR用 Python 程序 仿制这些请求形成模块封装一个 CliCLI 的假想模式>>> dp = dupan(user="neoney", bduss="")
>>> dp.list('/')
>>> dp.stat('/manga')
>>> dp.cd('/manga')
>>> dp.mkdir('/manga/arakawa')
>>> dp.cp('/manga/hagaren.zip', '/manga/arakawa/hagaren.zip')
>>> dp.move('/manga/hagaren.zip', '/manga/arakawa/hagaren.zip')
>>> dp.delete('/manga/hagaren.zip')
>>> dp.share('/manga/arakawa/hagaren.zip')
>>> dp.sharings
>>>
Package: dup-crypto
Duniter Protocol Crypto PythonDUP-crypto (Duniter protocol cryptography) binding from pure Rust to Python3.Installpip3 install --user --upgrade dup_cryptoBuild & installsudo pip3 install --upgrade setuptools wheel
sh build_wheel.sh
pip3 install --user --upgrade target/wheel/dist/dup_crypto-*-py3-none-any.whlNo need to install any additional crypto package!
Package: dupdict-mod
A Pure Python dictionary wrapper that allows duplicate keys is provided.It passes pylint, passes pep8, is thoroughly unit tested, and runs on CPython 3.3 - 3.10,
Pypy3 3.9 7.3.9.It can wrap a python dict(),https://pypi.org/project/treap/orhttps://pypi.org/project/red-black-tree-mod/- among other
things.
Package: dupecheck
DupecheckAnalyze your generated text for plagiarizing your datasetUse this to prevent"Thomas Riker" situations, when generating text with machine learning systems.AtEclectic Beams, when generating stories from finetuned GPT-2 models, I kept finding nice stories, but whole segments of them were just copied verbatim from the dataset text. I started battling this by using my editor's "find" feature on each line of generated text, or random subsequences against the dataset. It was time-consuming, and caused much eye pain and frustration, each time I'd read a story I liked, only to find out it had large sections copied verbatim.Time to automate the tedious away.Dupecheck will search for substrings of given length in a larger text dataset for you.
Use this to validate uniqueness before saving generated text, and you'll save yourself a huge headache.It compares word series without punctuation.pipinstalldupecheckfromdupecheckimportchunks,sliding_window,sliding_window2Chunking methodFind word series at least 5 words long in dataset.
(Prone to mid-pattern window-splitting false negatives, since the window doesn't slide. Your pattern might get cut and split between two chunks, then not found even if it should match)duplicates=chunks.dupecheck(min=5,max=10,text=my_gen_text,dataset=my_dataset,verbose=False)Sliding windowfind, try 1find word series at least 5 words long of your text string in dataset stringduplicates=sliding_window.dupecheck(min=5,max=10,text=my_gen_text,dataset=my_dataset,)Sliding windowfindtry 2find word series at least 5 words long of your text string in dataset string.cleans text and dataset for you if you don't provide cleaned and specifycleaned=True.Cleaning is currently VERY slow.duplicates=sliding_window2.dupecheck(min=5,max=10,text=my_gen_text,dataset=my_dataset,cleaned=False,verbose=False)Preprocessing helpersfromdupecheck.preprocessimportpre_process_textdataset=pre_process_text(dataset)text=pre_process_text(text)
Package: dupechecker
dupecheckerCheck for and delete duplicate files from the command line.InstallationInstall with:pip install dupecheckerUsage>dupechecker -h

usage: dupechecker [-h] [-r] [-i [IGNORES ...]] [-d] [-ad] [-ns] [paths ...]

positional arguments:
  paths                 The paths to compare files in.

options:
  -h, --help            show this help message and exit
  -r, --recursive       Glob files to compare recursively.
  -i [IGNORES ...], --ignores [IGNORES ...]
                        Ignore files matching these patterns. e.g. `dupechecker -i *.wav` will compare all files in the current working directory except .wav files.
  -d, --delete_dupes    After finding duplicates, delete all but one copy. For each set of duplicates, the tool will ask you to enter the number corresponding to the copy you want to keep. Pressing 'enter' without entering a number will skip that set without deleting anything.
  -ad, --autodelete     Automatically decide which file to keep and which to delete from each set of duplicate files instead of asking which to keep.
  -ns, --no_show        Don't show printout of matching files.
Package: dupe-remove
Welcome todupe_removeDocumentationHow come duplicate data in database?In OLAP database Redshift, theprimary_keycolumn doesn’t apply any restriction due to performance issue. What if our ETL pipeline load duplicate same data multiple times in retry?Howdupe_removesolve the problem?dupe_removeuse a optimized strategy to remove duplicate precisely and fast. You only need to specify:database connectiontable name, id column, sort key columndupe_removewill do these on your own will:remove duplicate data in specified sort key rangedeploy as cron job on AWS Lambda to automatically remove all duplicate data in a table.Usage ExampleOur database:table.events
|-- column(id, type=string)         # id column
|-- column(time, type=timestamp)    # sort key column
|-- other columns ...On Local Machinefromdatetimeimportdatetime,timedeltafromsqlalchemy_mateimportEngineCreatorfromdupe_removeimportWorkertable_name="events"id_col_name="id"sort_col_name="time"credential_file="/Users/admin/db.json"engine_creator=EngineCreator.from_json(credential_file)engine=engine_creator.create_redshift()worker=Worker(engine=engine,table_name=table_name,id_col_name=id_col_name,sort_col_name=sort_col_name,)worker.remove_duplicate(lower=datetime(2018,1,1),upper=datetime(2018,2,1),)On AWS Lambda Cron Jobdefhandler(event,context):fromdatetimeimportdatetime,timedeltafromsqlalchemy_mateimportEngineCreatorfromdupe_removeimportScheduler,Worker,Handlertable_name="events"id_col_name="id"sort_col_name="time"engine_creator=EngineCreator.from_env(prefix="DEV_DB",kms_decrypt=True)engine=engine_creator.create_redshift()test_connection(engine,6)worker=Worker(engine=engine,table_name=table_name,id_col_name=id_col_name,sort_col_name=sort_col_name,)# run every 5 min, clean 31 days data at a time from 2018-01-01,# start over in 12 cyclecron_freq_in_seconds=300start=datetime(2018,1,1)delta=timedelta(days=31)bin_size=12scheduler=Scheduler(cron_freq_in_seconds=cron_freq_in_seconds,start=start,delta=delta,bin_size=bin_size,)real_handler=Handler(worker=worker,scheduler=scheduler)real_handler.handler(event,context)Installdupe_removeis released on PyPI, so all you need is:$pipinstalldupe_removeTo upgrade to latest version:$pipinstall--upgradedupe_remove
Package: dupesearch
dupesearchA fast and simple command line tool written in rust and python for finding duplicate files.Should I use it?Probably not, this is still a WIP, it's quite messy, I haven't tested most of it, and was my first time using Rust so I probably did some dodgy stuff.How do I use it?Install thedupesearchpackage from pip using your preferred command.Runpython -m dupesearch -iand follow the instructions.ContributionsTo set up locally:Clone the repository withgit clone https://github.com/wookie184/dupesearch.gitcdinto the repository withcd dupesearchCreateandactivatea venv, and make sure to point your editor to the executable if necessary.Install the dev requirements withpip install -r requirements-dev.txtInstall the pre-commit hooks withpre-commit installInstall the rust crate as a module withmaturin develop --releaseFinally, run the project withpython -m dupesearchIf you'd like to make a contribution, feel free to and i'll try to merge it in. If it's not a tiny change please create an issue first, at least to ensure that i'm still active and able to review the changes.Any issues suggesting improvements are also welcome, although this was just done as a practice project to get started with rust and packaging python projects, so I may not implement every suggestion.
Package: dupeutil
dupeutilA command-line program written in Python for detecting and removing duplicate filesTable of contentsInstallationUsageArgumentsFiles flagQuiet flagRecursive flagInstallationTo installdupeutil, enter the following command in your terminal:python -m pip install dupeutilUsagedupeutilis designed to be as effortless and as easy as possible to use. You can check for duplicate files in a directory bycd-ing into that directory and by simply entering the following command in your terminal:dupeutilThat's it. The program will then check for duplicate files in thecurrent working directoryand provide you with a prompt that shows which duplicates were found and ask if you would like to remove them.Alternatively, if you would like to check a particular directory without it being the current working directory, you can explicitly pass it a directory:dupeutil <path to dir>📝 NOTE: You may need to wrap the directory in quotes:

dupeutil "C:\Users\user\Downloads\folder"ArgumentsThese are the available command-line arguments that you can pass todupeutil. You can enterdupeutil -hordupeutil --helpin your terminal to view them as well.positional arguments:
  dir                           the directory to check

options:
  -h, --help                    show this help message and exit
  -f FILES [FILES ...],         the files to compare
  --files FILES [FILES ...]

  -q, --quiet                   suppress output to the console; this option will not ask for
                                permission to  remove files
  -r, --recursive               include files in subdirectories
  -v, --version                 print program versionFiles flagIf you'd rather compare files instead of a directory, you can do so by using the-for--filesflag:dupeutil -f <path to file1> <path to file2> <path to file3> ... <path to fileN>If any of the paths contain a space, it's recommended that you wrap them in quotes.Quiet flagIf you don't want to see any output in the console, you can pass the-qor--quietflag when you rundupeutil:dupeutil <path to dir> -qIf you include the quiet flag, the program will not provide you with an overview of duplicate files, nor will it ask you for permission to remove those files.Recursive flagIf you would like to include subdirectories in the given directory, pass the-ror--recursiveflag:dupeutil <path to dir> -rIncluding this flag will cause files in the top-level directory and files in any subdirectories to be compared as well.
Package: dupfilefind
Find files with identical contents. Optionally hard-link or delete such files. Works on Windows. Can print out the md5sums and sizes of all your files. Unlike most tools of its ilk, this one will work even if the list of the metadata of all your files is too large to fit into your RAM.Dupfilefind is reasonably efficient, for what it does. It first compares sizes (so it can tell if files are different) and then inode numbers (so it can tell if different filenames are actually links to the same underlying file contents) before it resorts to computing MD5 sums of file contents. In addition I did some profiling and benchmarking to see how I could make it most efficient, and this is what I came up with.LICENCEYou may use this package under the GNU General Public License, version 2 or, at your option, any later version.  You may use this package under the Transitive Grace Period Public Licence, version 1.0, or at your option, any later version. (You may choose to use this package under the terms of either licence, at your option.)  See the fileCOPYING.GPLfor the terms of the GNU General Public License, version 2.  See the fileCOPYING.TGPPL.htmlfor the terms of the Transitive Grace Period Public Licence, version 1.0.SeeTGPPL.PDFfor why the TGPPL exists, graphically illustrated in four slides.
Package: dupfileremover
dupfileremoverdupfileremover is a python command line utility that takes in files from folders,
or individual file names, and hashes the files found to see if there are any matches.
This can be useful to see if you have any duplicates files that you may want to clean up.Basic Usagedupfileremover [OPTIONS] [FOLDERS]…Options:-n,--numberINTEGERThe maximum number of each file that should be found
(default is 1)--helpShow this message and exit.Examplesdupfileremover myPicsFolder/This command will hash every file within themyPicsFolder/and all sub folders
then look for duplicates. The default is set to 1. As in there should be only 1 unique
file in the directory.dupfileremover-n2 desktopWallpapers/videoGames/ desktopWallpapers/cars/This command looks in two different directories (videoGames and cars) that are both
within a directory calleddesktopWallpapers. Each files in both directories will be
hashed then compared to see if any matching files are found either in the same directory
or in the opposite directory. This command however uses the-nflag followed by the
number 2. This means that you know for a fact, or a fine if there is at least
one duplicate file found. If there are more than 2 duplicates found then it will be displayed
in the results.dupfileremover cat.jpeg picsOfCats/This command will hash the individualcat.jpegfile and all the files within thepicsOfCats/directory and compare all files for matches. This means you will be quickly
able to see ifcat.jpegalready exists withinpicsOfCats/with the same, or different, name.Prerequisitesdupfileremover requiresClick.Clickis what manages all arguments and options that are
passed to the program. It can be installed withpip install Click.LicenseThis project is licensed under the MIT License - See the LICENSE.md file for more details
Package: dupfilter
简介去重过滤器,提供常见的去重方案，开发便捷、性能极高。去重方案种类去重方案说明特点缺点置出方案MemoryMemoryFilter基于内存集合类型实现准确性高不能持久化随机删除FileFileFiler基于文件+集合类型实现准确性高本地内存和存储占用大利用文件指针区间删除RedisRedisBloomFilterAsyncRedisBloomFilter基于Redis Bitmap和布隆过滤器算法实现占用内存极小有误判的情况且不容易删除元素随机删除RedisStringFilterAsyncRedisStringFilter基于Redis String数据结构实现不会误判，能基于过期时间实现查询去重和确认机制占用资源很大，需尽可能压缩和设置过期时间设置过期时间RedisSetFilterAsyncRedisSetFilter基于Redis Set数据结构实现准确性高占用资源较大随机删除RedisSortedSetFilterAsyncRedisSortedSetFilter基于Redis SortedSet数据结构实现准确性高占用资源较大根据分值删除项目特点多种方案提供不同场景需求。基于Lua脚本支持批量操作，速度快。支持异步，可快速集成到异步代码和异步框架中。代码示例RedisBloomFilterimportredisfromdupfilterimportRedisBloomFilterserver=redis.Redis(host="127.0.0.1",port=6379)rbf=RedisBloomFilter(server=server,key="bf",block_num=2)print(rbf.exists_many(["1","2","3"]))rbf.insert_many(["1","2","3"])print(rbf.exists_many(["1","2","3"]))AsyncRedisBloomFilterimportasyncioimportaioredisfromdupfilterimportAsyncRedisBloomFilterasyncdeftest():server=aioredis.from_url('redis://127.0.0.1:6379/0')arbf=AsyncRedisBloomFilter(server,key='bf')stats=awaitarbf.exists_many(["1","2","3"])print(stats)awaitarbf.insert_many(["1","2","3"])stats=awaitarbf.exists_many(["1","2","3"])print(stats)loop=asyncio.get_event_loop()loop.run_until_complete(test())Others和上述示例类似关于作者邮箱：1194542196@qq.com微信：hu1194542196
Package: dupgee
dupgeeDupgee is a mini web framework developed for micro-python(Tested on esp8266).InstallationpipinstalldupgeeCreate ProjectdupgeecreatenewprojectLet's code inpages.pyand add view tourls.py.
When project has created, example views and urls already exists.
EditWIFI_SSIDandWIFI_PASSWORDvariables innewproject/runner.pyfile.RunThis operation needsadafruit-ampypackage. See alsohttps://learn.adafruit.com/micropython-basics-load-files-and-run-code/install-ampyinstallation and other details.Move filesampy--port/dev/<your-serial-port>putnewproject/Run
In host machineampy--port/dev/<your-serial-port>runnewproject/runner.py# or another portampy--port/dev/<your-serial-port>runnewproject/runner.py8000In serial connected python interpreterexec(open("./newproject/runner.py").read(),globals())TestContributingSee;https://github.com/ahmetkotan/dupgee
Package: dup-hawk
dup_hawk🦅 A bot the looks for duplicate issues on a GitHub repo, created especially for competitive security reviews.Getting StartedRequirementsgitYou'll know you did it right if you can rungit --versionand you see a response likegit version x.x.xPythonYou'll know you've installed python right if you can run:python --versionorpython3 --versionand get an output like:Python x.x.xpipxpipxis different frompipYou may have to close and re-open your terminalYou'll know you've installed it right if you can run:pipx --versionand see something likex.x.x.xpoetryYou'll know you've installed it right if you can runpoetry --versionand get an output likePoetry (version 1.4.2)Installationpipxpipx install dup_hawkFrom Sourcegit clone https://github.com/Cyfrin/dup-hawk
cd dup-hawk
pip install . -eQuickstartdup-hawk --helpTo run the duplicate checker, run:dup-hawk --git-repo-url {URL} --git-pat-token {PAT_TOKEN}And you will have marked the issues on your github repo!Testspoetry run pytestIntegration testspoetry run pytest -m ""Specific tests with pdb and stdoutpoetry run pytest -k "test_mark_duplicates_from_dfs" --pdb -sBig TODOAdd mocking unit tests.
Package: duphunter
# DuphunterDuphunter is a very simple application that finds and lets you delete duplicate files interactively, using a very efficient list interface where you can mark files for preservation or deletion, and later commit your changes.  The scanning process should scale well into the millions of files.The repository, bug tracker and Web site for this tool is at [http://github.com/Rudd-O/duphunter](http://github.com/Rudd-O/duphunter).## Setting upInstall PyQt5Install this package usingpip,python setup.py installorpython setup.py bdist_rpm## RunningRun the commandduphunteron the command line, optionally specifying paths to scan as arguments to the command.
Package: dupin
Welcome todupina Python package for detecting rare events in molecular simulations.Overviewdupinis designed to provide an unopinionated Python API for partitioning temporal point cloud data into regions of stability and transition.
Generally such data comes from molecular simulations or experimental imaging techniques.
For example, if a researcher imaged gold nanoparticles nucleating into FCC crystal,dupincould help to partition the system into the initial fluid, transition, and crystal regions of the point cloud trajectory.
Though,dupinattempts to be general andunopinionatedwe provide sensible defaults and do not sacrifice ease of use for customizability.ResourcesDocumentation: Read our API documentation.Installation Guide: Look at our installation guide.GitHub Repository: Check out the source code.Issue Tracker: File a bug report or pull request.Related ToolsHOOMD-blue: Molecular simulation enginefreud: Molecular trajectory analysisruptures: Change point decection librarykneed: Elbow detection libraryExampleimportdupinasduimportnumpyasnpimportrupturesasrptsignal=np.load("signal.npy")dynp=rpt.Dynp(custom_cost=du.detect.CostLinearFit())detector=du.detect.SweepDetector(dynp,8)chps=detector.fit(signal)CreditsThis package was created withCookiecutterbased on a
modified version ofaudreyr/cookiecutter-pypackage.
Package: dupla
Project descriptionA Python API forDUPLA(Dataudvekslingsplatformen)A list of the services available can be foundhereSetupFor developmentpython -m venv .venvsource .venv/bin/activatepip install ".[test]"Run tests:pytest tests/Getting StartedUsing pre-defined API endpointsThis package defines some endpoints which has some (limited) knowledge
of the API schema used by SKAT. The classes are:dupla.payload.KtrPayloaddupla.payload.LigPayloaddupla.payload.MomsPayloaddupla.payload.KtrObsPayloaddupla.payload.LonsumPayloaddupla.payload.SelskabSambeskatningPayloaddupla.payload.SelskabSelvangivelsePayloadPlease c.f. the docs of the respective classes for more information
on what each endpoint is for. Each payload model is a Pydantic model, so inputs
are validated.Using the base APIThe base api access class can be used as following.
In this example, we access the VAT (Moms in Danish) endpoint.fromdatetimeimportdate,timedeltafromuuidimportuuid4importduplafromduplaimportDuplaApiKeysapi=dupla.DuplaAccess(transaction_id=str(uuid4()),agreement_id="your-aftale-id-goes-here",pkcs12_filename="path-to-cert-file",pkcs12_password="goodpassword",billetautomat_url="https://oces.billetautomat.skat.dk/auth/realms/oces/certificates/cert",jwt_token_expiration_overlap=5)# lets see if this company (se_number 98765432) has done any VAT the last yearkwargs={DuplaApiKeys.SE:["98765432"],DuplaApiKeys.AFREGNING_START:date.today()-timedelta(days=365),DuplaApiKeys.AFREGNING_SLUT:date.today()}payload=dupla.payload.MomsPayload(**kwargs)data=api.get_data(payload)print(data)© ERST 2023
Package: duplicate-detector
DUPLICATE-DETECTORFunctions:$python-mduplicate-detector[compare|sort]Compare :$python-mduplicate-detectorcompareINPUT_DIRTARGETScan both directories (INPUT-DIRandTARGET). Compares the files solely onFILE NAME. Every file found in both directories are moved from their originals paths inINPUT-DIRintoINPUT-DIR/DUPLICATES.Files in theTARGETdirectory are never moved.sort:$python-mduplicate-detectorsortINPUT_DIROUTPUT_DIRSort a directoryINPUT-DIRof pictures and videos ('JPG', 'JPEG', 'PNG', 'MP4', 'MOV') into anOUTPUT-DIR. Files inOUTPUT-DIRare sorted into folders by year and month, based on their metadata, as following :OUTPUT-DIR/YEAR/MONTH/pictures.If multiple file have the samefile name, the heaviest get sorted intoOUTPUT-DIR/YEAR/MONTHand the others intoOUTPUT-DIR/DUPLICATES.If no metadata is found for a file, it is not moved.
If a file is not in the supported extensions, it is not moved.Installation:For ubuntu:Install python3.8 , and required librairies:Without virtual env:$sudopython3.8-mpipinstall-rrequirements.txtWith virtual env:$virtualenvvenv-p/usr/bin/python3.8
$sourcevenv/bin/activate
$python-mpipinstall-rrequirements.txt
Package: duplicate-image-finder
Duplicate Image FinderDuplicate image finder uses image hashing to find similar/duplicate images in your local storage. All you gotta do isinstall,run (will setup the database with table) if no configuration is provided,run specifying which directory to look for images, and finallyrun asking it to show duplicate/similar images.Please note that it is a prototype. Please use at your own discretion.For example:# 1. installingpython3.9-mpipinstall--userduplicate-image-finder# 2. show helpduplicate-image-finder--help# 3. add directory images and calculate hashes using 4 threadsduplicate-image-finder--add<directory>--parallel4# 4. show the duplicate/similar images found in your browserduplicate-image-finder--showRunning 4 will result in opening a browser that shows duplicate/similar images. If you click on delete, it will be moved to .Trash folder.RequirementsLots, but all of them can be installed as dependencies as long as you are usingpython3.9. Unfortunately, some of its dependencies have not been made available inpython3.10yet, so we are stuck there.PoetryInstalling dependenciespoetryinstallRunningpoetryrunpythonduplicate_image_finder/duplicate_finder.py--showTestingpoetryrunpytestetc.This duplicate image finder source code is inspired/partially copied fromhttps://github.com/philipbl/duplicate-images.git.Significant changes from the referred version are:moved frommongodbtosqliteIs probably better in terms of finding similar images (or perhaps I misunderstood the previous code)Concepts/Technologies I learned/tried to learn while doing this:poetryfor dependencypytestfor unit testpysqlite3for databaseconcurrencyfor performanceimagehashfor perpetual image hashing for finding similaritygrouping CLI arguments in python (mutually exclusive, etc) usingargparser