Saravanan666/pypi-org huggingface dataset:
Package: dltn-checker
AboutTests whether records from an OAI-PMH feed pass minimum requirements of DLTN and optionally harvests only the good
records from a request to disk so that they can be added to Repox and included in the DPLA.InstallRunning with Builtin Argument Parsing from a CLIIf you want to do it this way, you’re going to need to clone this.  It’s also suggested to  build this with pipenv.$gitclonehttps://github.com/DigitalLibraryofTennessee/check_and_harvest$cdcheck_and_harvest$pipenvinstall$pipenvshellUsing OAIChecker from the dltnchecker moduleIf you’re cool :sunglasses: :$pipenvinstalldltn_checkerOtherwise:$pipinstalldltn_checkerExamples with the Built In Argument ParserCheck for bad DC records in an entire OAI-PMH feed.$pythonrun-ehttp://my-oai-endpoint:8080/OAIHandler-moai_dcCheck and harvest good DC records from an entire OAI-PMH feed.$pythonrun-ehttp://my-oai-endpoint:8080/OAIHandler-moai_dc-HTrueCheck and harvest good xoai records from a specifc set.$pythonrun-ehttp://my-oai-endpoint:8080/OAIHandler-mxoai-smy_awesome_xoai_set-HTrueCheck and harvest good MODS records from an entire provider in Repox.$pythonrun-ehttp://my-oai-endpoint:8080/OAIHandler-mMODS-pCrossroadstoFreedomr0-HTrueExamples using the OAIChecker Class from dltncheckerCheck a set to see if there are any bad files in a set.fromdltnchecker.harvestimportOAICheckerrequest=OAIChecker("https://dpla.lib.utk.edu/repox/OAIHandler","crossroads_sanitation","MODS")request.list_records()print(request.bad_records)By default, this will try to download the good files to a directory called output. If you don’t want to download, you
need to pass an additional parameter called harvest and set to False.fromdltnchecker.harvestimportOAICheckerrequest=OAIChecker("https://dpla.lib.utk.edu/repox/OAIHandler","crossroads_sanitation","MODS",harvest=False)request.list_records()print(request.bad_records)
Package: dl-toolkits
DL Toolkits:man_technologist:  Analytical tools for deep learning experiments  :woman_technologist:Whenever I analyzed the results of the DL experiment, I had to re-implement the analysis function every time. So, I implement frequently used functions in this repository. New features continue to be implemented, and simple examples of function usages can be found in theexamplesdirectory.InstallationYou can install the package withpipcommand. Python>=3 are supported.pip install dl-toolkitsYou can check the version of the package using the following commands.importtoolkitsprint(toolkits.__version__)ModulesVisualizationviz.tsne: t-SNE plotClustering qualitycluster.sse: Sum of squared error(SSE) [^1]cluster.batch_sse: Sum of squared error(SSE) for batch inputcluster.nsse: SSE normalized by the squared distance to the nearest interfering centroid(nSSE) [^1]cluster.batch_nsse: SSE normalized by the squared distance to the nearest interfering centroid(nSSE) for batch inputcluster.nearc: Top N nearest interfering centroidcluster.rfc: Feature space clustering quality(R_fc) [^2]Linear algebralinalg.get_singular_values: Get singular values per classlinalg.get_sum_of_singular_values: Get sum of singular values per classlinalg.get_average_sum_of_singular_values: Get average of sum of singular values per class [^3]Pretty printpprint.pred_summary: Simple print for predictions and true labelsLog parserparse.between_lines: Extract the log between the two input sentencesparse.between_lines_on_file: Extract the log between the two input sentences on the target fileparse.between_lines_on_dir: Extract the log between the two input sentences on the target directoryPyTorch helper functiontorch_helper.freeze_selected_param: Freeze the weights with the selected nametorch_helper.get_important_param_idx: Get important parameters indices[^4]References[^1]: Yoon, Sung Whan, et al. "Xtarnet: Learning to extract task-adaptive representation for incremental few-shot learning."International Conference on Machine Learning. PMLR, 2020.
[^2]: Goldblum, Micah, et al. "Unraveling meta-learning: Understanding feature representations for few-shot tasks."International Conference on Machine Learning. PMLR, 2020.
[^3]: Verma, Vikas, et al. "Manifold mixup: Better representations by interpolating hidden states."International Conference on Machine Learning. PMLR, 2019.
[^4]: Mazumder, Pratik, Pravendra Singh, and Piyush Rai. "Few-Shot Lifelong Learning."Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 3. 2021.Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python."the Journal of machine Learning research12 (2011): 2825-2830.
Package: dlt-personio-source
dlt-personio-sourceParent tables'employees', 
'absences', 
'absence_types', 
'attendances'some of these tables have sub-tablesto join the parent table to the sub table, use the joinparent.dlt_id = child.parent_dlt_idUsageinstall librarypipx install dlt-personio-sourceif the library cannot be found, ensure you have the required python version as per thepyproject.tomlfile.Run the source as below to load a sample data set.Add credentials and remove thedummy_dataflag to enable loading your data.First, import the loading method and add your credentialsfrom dlt_personio_source import load_personio_tables

#target credentials
# example for bigquery
creds = {
  "type": "service_account",
  "project_id": "zinc-mantra-353207",
  "private_key_id": "example",
  "private_key": "",
  "client_email": "example@zinc-mantra-353207.iam.gserviceaccount.com",
  "client_id": "100909481823688180493"}
  
# or example for redshift:
# creds = ["redshift", "database_name", "schema_name", "user_name", "host", "password"]

#Personio credentials
#get credentials at this url - replace"test-1" with your org name
#https://test-1.personio.de/configuration/api/credentials/management
client_id = ''
client_secret = ''then, you can use the code below to do a serial load:# remove some tables from this list of you only want some endpoints
tables = ['employees', 'absences', 'absence_types', 'attendances']
load_personio_tables(client_id=client_id,
                     client_secret=client_secret,
                     target_credentials=creds,
                     tables=tables,
                     schema_name='personio_raw',
                     dummy_data=True)or, for parallel load, create airflow tasks for each table like so:tables = ['employees', 'absences', 'absence_types', 'attendances']
for table in tables:
    load_personio_tables(client_id='',
                         client_secret='',
                         target_credentials=creds,
                         tables = [table],
                         schema_name='personio_raw',
                         dummy_data = True)If you want to do your own pipeline or consume the source differently:from dlt_personio_source import PersonioSource, PersonioSourceDummy

prod = PersonioSource(client_id='',
              client_secret='')
              
dummy = PersonioSourceDummy()

sample_data = dummy.tasks() 

for task in tasks:
    print(task['table_name'])
    for row in task['data']
        print(row)
Package: dlt-pipedrive-source
dlt-pipedrive-sourceprototype for source creationParent tables'deal', 'deal_participant', 'deal_flow'Usageoptionally Create a virtual environmentpython3 -m venv ./dlt_pipedrive_env
source ./dlt_pipedrive_env/bin/activateinstall librarypip install dlt-pipedrive-sourceIf the library cannot be found, ensure you have the required python version as per thepyproject.tomlfile.
(3.8+)You can run the snippet file below to load a sample data set.
You would need to add your target credentials first.
copy the file below and add credentials, then run itrun_load.pypython run_load.pyIf you want only some of the tables, pass them as an argument in the tables list in the loader (see below)# of all tables, get only deal_participant
tables = ['deal_participant'] 
# get all tables
tables = [] 

You can also toggle "mock data" to True in the load function, or pass None credentials, to try mock sample data.
Package: dltpy
dltpy - library and utilities for DLT loggingInstallingRequirements (not handled by pip/setuptools)native build environment:python3-dev,cmakeand adecent c++ compiler supporting c++17.fmtlibfor native logging and formatting exceptionsInstallationOnce the requirements mentioned above are ready, you can install dltpy with pip just as any other python package:current master from git:pip install --user git+https://github.com/Equidamoid/dltpy.gitlast version from pypi:pip install --user dltpyan editable "developer" mode:git clone https://github.com/Equidamoid/dltpy.git
pip install --user -e dltpyUsingThere are 3 commandline utilities that demonstrate usage of the library:dltpy-printprints specified .dlt file as plain text. "text-like" payloads are decoded and printed as text, others arerepr()eddltpy-filterfilters out unneeded boring messages. Run it with-f IDS:OF YOUR:APPSand enjoy tiny logs that take seconds to load.dltpy-receivedemonstrates integration with asyncio and direct use of the native module, likedlt-receive. The main difference is that it gracefully reconnects if, for example, your device reboots.Please note that non-verbosemessages are skipped byDltFilesince it has no way of decoding the payload.
Package: dltraffic
python homeworkDisplay download traffic from the Internet in various countries around the world.
Show the number of pc game users by country.
by using data form https://store.steampowered.com/stats/content/dltrafficneed import pandas as pd and import matplotlib.pyplot as plt
python_requires='>=3.6'
MIT LicenseHow to install$ pip install dltrafficHow to run$ dltrafficexample
Package: dltreader
ThedltreaderPython module allows the reading of AUTOSAR
[“Diagnostic, Log and Trace Protocol”](https://www.autosar.org/fileadmin/standards/foundation/1-0/AUTOSAR_PRS_DiagnosticLogAndTraceProtocol.pdf) files.
files.Why?The goal of this module is to provide a simple, purely Python-based module to
read.dltfiles without further dependencies. Apart from that a permissive
free software license shall is supported to make it usable in possible
commercial contexts.Examplefrom dlt import DltReader

with DltReader("example.dlt") as r:
    for storage_header, packet in r:
        if not packet.has_payload():
            # skip packets that do not have a payload
            continue

        print(packet)LimitationsAt current point in time the module is a PoC so should not be used in
production software; besides, it is only a partial implementation of the
overall standard and not optimized for performance.DocsPlease check thedocs/directory for further details.Related DLT projectsPython modules for DLT (without claim to be complete):https://pypi.org/project/dltpy(requires C++ dependencies)https://github.com/r2b1d1/dlt2json(converts DLT to json format)https://gitlab.com/Menschel/logging-dlt(similar but using GPL)https://pypi.org/project/pydlt(provides also writing of DLT messages)…Similar projects in other programming languages:https://github.com/esrlabs/dlt-reader/https://github.com/COVESA/dlt-daemon/…
Package: dlt-sidestep
Installationpip install in your Databricks Notebook%pipinstalldlt_sidestepExample UsageNote: You must define apipeline_idvariable asspark.conf.get("pipelines.id", None)Note: You must define agvariable asglobals()`frompyspark.sql.functionsimport*frompyspark.sql.typesimport*fromdlt_sidestepimportSideSteppipeline_id=spark.conf.get("pipelines.id",None)g=globals()ifpipeline_id:importdltjson_path="/databricks-datasets/wikipedia-datasets/data-001/clickstream/raw-uncompressed-json/2015_2_clickstream.json"step="""@dlt.create_table(comment="The raw wikipedia click stream dataset, ingested from /databricks-datasets.",table_properties={"quality": "bronze"})def clickstream_raw():return (spark.read.option("inferSchema", "true").json(json_path))"""SideStep(step,pipeline_id,g)df=clickstream_raw()df.display()step="""@dlt.create_table(comment="Wikipedia clickstream dataset with cleaned-up datatypes / column names and quality expectations.",table_properties={"quality": "silver"})@dlt.expect("valid_current_page", "current_page_id IS NOT NULL AND current_page_title IS NOT NULL")@dlt.expect_or_fail("valid_count", "click_count > 0")def clickstream_clean():return (dlt.read("clickstream_raw").withColumn("current_page_id", expr("CAST(curr_id AS INT)")).withColumn("click_count", expr("CAST(n AS INT)")).withColumn("previous_page_id", expr("CAST(prev_id AS INT)")).withColumnRenamed("curr_title", "current_page_title").withColumnRenamed("prev_title", "previous_page_title").select("current_page_id", "current_page_title", "click_count", "previous_page_id", "previous_page_title"))"""SideStep(step,pipeline_id,g)df=clickstream_clean()df.display()
Package: dlup
Deep Learning Utilities for PathologyDlup offers a set of utilities to ease the process of running Deep Learning algorithms on
Whole Slide Images.FeaturesRead whole-slide images at any arbitrary resolution by seamlessly interpolating between the pyramidal levelsDataset classes to handle whole-slide images in a tile-by-tile manner compatible with pytorchAnnotation classes which can load GeoJSON,V7 Darwin,HALOandASAPformats and read parts of it (e.g. a tile)Transforms to handle annotations per tile, resulting, together with the dataset classes a dataset consisting of tiles of whole-slide images with corresponding masks as targets, readily useable with a pytorch dataloaderCommand-line utilities to report on the metadata of WSIs, and convert masks to polygonsCheck thefull documentationfor more details on how to use dlup.QuickstartThe package can be installed usingpython -m pip install dlup.Used byahcore: a pytorch lightning based-library for computational pathologyCiting DLUPIf you use DLUP in your research, please use the following BiBTeX entry:@software{dlup,
  author = {Teuwen, J., Romor, L., Pai, A., Schirris, Y., Marcus, E.},
  month = {12},
  title = {{DLUP: Deep Learning Utilities for Pathology}},
  url = {https://github.com/NKI-AI/dlup},
  version = {0.3.36},
  year = {2023}
}or the following plain bibliography:Teuwen, J., Romor, L., Pai, A., Schirris, Y., Marcus E. (2023). DLUP: Deep Learning Utilities for Pathology (Version 0.3.36) [Computer software]. https://github.com/NKI-AI/dlup
Package: dLux
∂LuxDifferentiable Optical Models asParameterised Neural Networksin Jax using ZodiaxContributors:Louis Desdoigts,Jordan Dennis,Adam Taras,Max Charles,Benjamin Pope,Peter Tuthill∂Lux is an open-source differentiable optical modelling framework harnessing the structural isomorphism between optical systems and neural networks, giving forwards models of optical systems asparametric neural networks.∂Lux is built inZodiax, which is an open-source object-orientedJaxframework built as an extension ofEquinoxfor scientific programming. This framework allows for the creation of complex optical systems involving many planes, phase and amplitude screens in each, and propagates between them in the Fraunhofer or Fresnel regimes. This enablesfast phase retrieval, image deconvolution, andhardware design in high dimensions. Because ∂Lux models are fully differentiable, you canoptimize them by gradient descent over millions of parameters; or useHamiltonian Monte Carlo to accelerate MCMC sampling. Our code is fully open-source under a 3-clause BSD license, and we encourage you to use it and build on it to solve problems in astronomy and beyond.The ∂Lux framework is built inZodiax, which gives it a deep range of capabilities from bothJaxandEquinox:Accelerated Numpy: a Numpy-like API that can run on GPU and TPUAutomatic Differentiation: Allows for optimisation and inference in extremely high-dimensional spacesJust-In-Time Compilation: Compiles code into XLA at runtime and optimising execution across hardwareAutomatic Vectorisation: Allows for simple parallelism across hardware and asynchronous executionFor an overview of these capabilities and different optimisation methods inZodiax, please go through thisZodiax Tutorial.Documentation:https://louisdesdoigts.github.io/dLux/Requires: Python 3.10+, Jax 0.4.13+, Zodiax 0.4+Installation:pip install dLuxIf you want to run the tutorials locally, you can install the 'extra' dependencies like so:pip install 'dLux[extras]'Collaboration & DevelopmentWe are always looking to collaborate and further develop this software! We have focused on flexibility and ease of development, so if you have a project you want to use ∂Lux for, but it currently does not have the required capabilities, have general questions, thoughts or ideas, don't hesitate toemail meor contact me ontwitter! More details about contributing can be found in ourcontributing guide.PublicationsWe have a multitude of publications in the pipeline using dLux, some built from our tutorials. To start we would recommend looking atthis invited talkon ∂Lux which gives a good overview and has an attached recording of it being presented! We also havethis poster!
Package: dLuxToliman
dLuxTolimanDescriptionThis repository/package contains pre-built ∂Lux models of the Toliman optical system, and pre-built parametrised ∂Lux source objects for Alpha Centauri.∂Luxis an open-source differentiable optical modelling framework harnessing the structural isomorphism between optical systems and neural networks, giving forwards models of optical system as aparametric neural network.
∂Lux is built inZodiaxwhich is an open-source object-orientedJaxframework built as an extension ofEquinoxfor scientific programming.The primary goal of the Toliman mission is to discover Earth-sized exoplanets orbiting in Alpha Centauri, the closest star system to our own.
To achieve this, the mission will employ a novel telescope design that will be able to detect subtle changes in the positions of the Alpha Centauri binary pair.
These changes are caused by the gravitational reflex motion induced by an Earth-sized companion, and this cutting-edge technology will enable scientists to identify exoplanets too small to be detected by conventional telescopes.
Toliman utilises a binary phase diffraction pupil to grasp the expected microarcsecond-scale astrometric signal.InstallationdLuxTolimanis pip-installable. To install the latest release, simply run:pipinstalldLuxTolimanUsageA very basic example of how to use this package is as follows:# importsimportdLuxTolimanasdlTfrommatplotlibimportpyplotaspltosys=dlT.TolimanOpticalSystem(psf_npixels=128)# creating Toliman optical systemsource=dlT.AlphaCen(n_wavels=3,separation=8,position_angle=30)# creating Alpha Centauri sourcepsf=osys.model(source)# running optical simulation# plottingplt.imshow(psf**.5,cmap='inferno')plt.title('Toliman $\sqrt{PSF}$')plt.show()ContributorsMax Charles,Louis Desdoigts,Benjamin Pope, andPeter Tuthill.LicenseThis package is released under the BSD 3-Clause License. See theLICENSEfile for more information.SupportIf you encounter any problems or have any questions related to Toliman models, please raise an issue or contact me atmax.charles@sydney.edu.au.
Package: dlvh
dlvhdlvh (downloadvideohere) is a wrapper foryt-dlpthat allows you to quickly download videos in a folder according to predefined nested local rules.InstallpipinstalldlvhHow to useUsedlvh *URL*to download to current folder,dlvh -p *preset* *URL*to download to a preset, anddlvh -n *preset*(in the target directory) to define new preset.You can have dlvh configuration files on each level of the directory tree. These files use a simple yaml structure:options:download-archive:.ytdlarchive# Relative to the folder where the download happenscookies:^/yt_cookies.txt# Relative to THIS configuration file, even in subfoldersoutput:'"%(id)s.%(title)s.%(ext)s"'# Careful with quoting strings!merge-output-format:mkvflags:-restrict-filenames-prefer-free-formatsLet's say you have the following folder structure:├── audio  
│   ├── dlvh.yml # `extract-audio`  
│   ├── unsettling_screaming  
│   └── calm_nature_sounds  
└── video  
    ├── dlvh.yml # `embed-thumbnail`, `download-archive: .ytdlarchive`  
    ├── asmr  
    └── dance_covers  
        ├── dlvh.yml # `download-archive: ^/ytdlarchive`  
        ├── twice  
        └── redvelvetWhen you download something toaudio/calm_nature_sounds, the flagextract-audiofrom its parent folder will be respected.When you download something tovideo/asmr, the flagembed-thumbnailand the optiondownload-archive: .ytdlarchivewill be respected, meaning a local filevideo/asmr/.ytdlarchivewill be used as archive.When you download something tovideos/dance_covers/twicehowever, its parent folder's optiondownload-archive: ^/ytdlarchiveoverwrites this, meaning a common archive filedance_covers/.ytdlarchivewill be used for all subfolders.
Package: dlvr
Email sending for humansInstallationwith pip as easy as:$ pip install dlvror checkout the latest version from github:$ git clone https://github.com/bmaeser/dlvr.git
$ cd dlvr
$ python setup.py installQuickstartopen a connection to a server:>>> from dlvr import SMTPServer
>>> s = SMTPServer()create a email:>>> from dlvr import Message
>>> m = Message('bob@example.com', ['alice@gmail.com', 'support@example.com'],
        'testsubject', 'testbody')send the email:>>> s.connect()
>>> s.send(m)
>>> s.disconnect()Full examplefrom dlvr import SMTPServer, Message

server = SMTPServer(host="smtp.googlemail.com", port='587',
    auth_user='MYUSERNAME', auth_pass='MYPASSWOR', tls=True)

## host (optional): defaults to localhost
## port (optional): defaults to 25
## auth_user (optional): your usernamer
## auth_pass (optional): your passwort
## tls (optional): encrypt the session defaults to False

text = 'here is you link: http://www.google.com'
subject = 'the link you asked for'

html = """\
<html>
    <head></head>
    <body>
        <p>Hi!<br>
        Here is the <a href="http://www.google.com">link to google</a> you wanted.
        </p>
    </body>
</html>
"""

message = Message('bob@example.com', ['alice@gmail.com', 'support@example.com'],
    subject, text, alternatives=[(html, 'text/html')])

## constructor arguments:

## from_email: required, the senders email
## to: required, a list of recipients
## subject: required, the emails subject
## text_message (optional): the text representation of the email body
## cc (optional): a list of the carbon-copy recipients
## bcc (optional): a list of blind-carbon-copy recipients
## attachments (optional): a list of attachments, and the mimetype to use eg:
##      attachments = [('/tmp/image.jpg', 'image/jpeg'), ('/tmp/song.mp3', 'audio/mpeg3')]
## alternatives (optional): a list of alternative representation of the email body
##      and the mimetype to use
## charset (optional): the charset/encoding to use for text_message, defaults to utf-8

## message functions:

# attach_alternative(content, 'mimetype')
# where mimetype is optional und defaults to 'text/html'

# attach_file('/path/to/file', 'mimetype')
# where mimetype is opional and is guessed if not provided

server.connect()
server.send(message)
## send another message with the same open connection ...
server.disconnect()

## or shorthand if you only send one message:
server.send_email(message)Contributepull-request please and/or create a issue on github