Saravanan666/pypi-org huggingface dataset:
Package: datanerd
stats(), iv_woe(), pushdb()This function provides some statistical summary of a given dataframe.Thestats()function takes in a dataframe and returns the following statistics:countmeanstdmin10th percentile20th percentile25th percentile30th percentile40th percentile50th percentile (median)60th percentile70th percentile75th percentile80th percentile90th percentile95th percentile99th percentilemax% of missing values% of non-zero values#numberofunique_valuesimportpandasaspdimportdatanerdasdndf=pd.read_csv('titanic.csv')dn.stats(df)Theiv_woe()function is used to calculate the Weight of Evidence (WoE) and Information Value (IV) for a given dataframe.The WoE is a measure of how much the presence or absence of a predictor (feature) contributes to the probability of target.The IV is a measure of the strength of the relationship between the predictor and the target.Theiv_woe()function takes in the following arguments:data:a dataframe containing the predictor variables and the target variabletarget:the name of the target variablebins:the number of bins to use for discretizing continuous variablesoptimize:a boolean indicating whether to optimize the binning of the continuous variablesthresold:the minimum percentage of non-events (negative outcome) in each bin for optimization. If optimize is set to True, the function will iterate over the number of bins from 20 to 1 and calculate the WoE and IV for each bin. If the percentage of non-events in each bin is greater than or equal to the specified thresold, it will return the WoE and IV for that bin. If it cannot find a binning that meets the thresold, it will return the WoE and IV for the best bin it could find.If optimize is set to False, the function will calculate the WoE and IV for the specified number of bins.The function returns a dataframe containing the WoE and IV for each predictor variable.importpandasaspdimportdatanerdasdndf=pd.read_csv('cancer.csv')iv,woe=dn.iv_woe(data=df,target='Diagnosis',bins=20,optimize=True,thresold=True)Thispushdb()function takes Pandas dataframe (data), a table name (tablename), a server name (server), a database name (database), and a schema name (schema)It then creates a connection to a Microsoft SQL Server database using these input parameters, and pushes thedatai.e dataframe to the specified table in the database. The function is written to handle a fast execution of many records using thefast_executemanyargument in thecreate_enginefunction.importpandasaspdimportdatanerdasdndf=pd.read_csv('day.csv')dn.pushdb(df,tablename='day',server='SQL-DW',database='schedule',schema='ana')This will push the data i.e dataframe to thedaytable in thescheduledatabase on theSQL-DWserver, using theanaschema.
Package: datanest
datanestA simple, intuitive, pandas-based database.Perfect for handling data such as time series, images, or any
Python objects alongside their metadata. This tool encapsulates a pandas
DataFrame containing metadata and Python objects. It provides an intuitive
data and metadata retrieval syntax through keyword-arguments.Installationpip install datanestUsagedatanest.Databaseis the core class that wraps apandas.DataFrameobject. Even before adding any data fields using theadd_data_fieldmethod, the database can already be used to query rows from the encapsulated DataFrame with an intuitive keyword argument syntax.importdatanest# Load example DataFrame with columns:# participant_id (int), age (float), surgery_performed (bool), notes (str)db=datanest.get_example_database()# Retrieve all metadatadb()# Retrieve metadata for participant 3db(participant_id=3)# Retrieve metadata for participants aged 50 to 60 who have not had surgerydb(age_lim=(50,60),surgery_performed=True)# Retrieve metadata for participants where the notes string contains the word interestingdb(notes_has='interesting')Theadd_data_fieldmethod can be used to add arbitrary python objects to the database, and we can retrieve relevant data entries using the same keyword argument syntax.# Add heart rate data to the database, indexed by participant_iddb.add_data_field('heart_rate',datanest.get_example_data(),'participant_id')# Retrieve all heart rate time series datadb.heart_rate()# Retrieve heart rate time series data for participant 3db.heart_rate(participant_id=3)# Retrieve heart rate time series for participants aged 50 to 60db.heart_rate(age_lim=(50,60))# Retrieve heart rate time series for participants where the notes string contains the word interestingdb.heart_rate(notes_has='interesting')Licensedatanestis distributed under the terms of theMIT license.ContactPraneeth NamburiProject Link:https://github.com/praneethnamburi/datanestAcknowledgmentsThis tool was developed as part of the ImmersionToolbox initiative at theMIT.nano Immersion Lab. Thanks to NCSOFT for supporting this initiative.
Package: datanews
Datanews PythonThis is a client library for python 3, built to
interact with the Datanews API. You can find full api documentation
atofficial documentation page.InstallUse pip to install the library.pipinstalldatanewsUsageTo use the API, import thedatanewspackage and provide the api key. Then, proceed to make
queries by calling corresponding functions of the library.Core APIimportdatanewsdatanews.api_key='API_KEY'response=datanews.headlines(q='SpaceX')articles=response['hits']print(articles[0]['title'])Monitoring APITo create monitors use thedatanews.Monitorobject.
To retrieve monitor runs useMonitor.latestfunction and passrun_id. Each
run object has alast_run_idfield, which refers to previous run, orNone,
if no runs were made previously (i.e. this was the first run). Please refer to
tofull documentationof monitoring API.importdatanewsdatanews.api_key='API_KEY'# list all monitorsprint(datanews.Monitor.list())# create new monitor, pass query (list of keywords), and webhook urlmonitor=datanews.Monitor.create(query='SpaceX','webhook'='https://mywebhook.com')# get id of the monitorid=monitor['id']# list latest articles from the Monitorarticles=[]run_id=monitor.get('last_run_id',None)done=run_idisNonewhilenotdone:run=datanews.Monitor.latest(run_id)articles.extend(run['articles'])run_id=run['last_run_id']done=run_idisNone
Package: datanexus
DataNexusDataNexus is a simple to use Python module that you can use in your projects to get transcripts, datasets, etc.
The module also allows you to extract character lines from transcripts witch makes it easyer for you to be able to do finetunning of a GPT2 model as an example.Key feacturesDownloading of Datasets and TranscriptsExtract Characters from TranscriptsInstallationTo get started:pip install datanexusUsage⚠️ | Full documenation link to come in the future and the code may be unstable as in testing!Downloading of Datasets/Transcriptsfromdatanexusimportdatanexusdatanexus=datanexus()datasets=datanexus.download_dataset('ironman.txt')print(datasets)Extract character's from transcriptsYou will need to create a folder calledModelsto sucessfully extract the character informationfromdatanexusimportdatanexuscharacter=datanexus.save_character('ironman.txt','Tony Stark:','Tony.txt')print(character)Showing all of the possible datasetsThis function will show all of the possible datasets that is usablefromdatanexusimportdatanexusdatasets=datanexus.possible_models()fordatasetindatasets:print(dataset)SupportIf you have any question or any issues then feel free to create an issue onGithub.Feel free to joinThe Workshopdiscord server and send me a ping (_Ethan_)
Package: datanodeclient
The OpenAPI specification implemented by NLP Sandbox Data Nodes. # Overview A NLP Sandbox Data Node is a repository of clinical notes that implements this OpenAPI specification so that other services in the NLP Sandbox ecosystem can access them. For example, a client requests data from a Data Node before passing them as input to an NLP Tool like a Date Annotator, Person Name Annotator, etc. For the sake of benchmarking NLP Tool, a Data Node can also give access to the gold standard that the NLP Tool is expected to infer (e.g. annotations).   # noqa: E501
Package: datanonymizer
Anonymizer tool for datasets such CSV files.To generate fake data, you can choose between two excelent generators:Faker(default).mimesisvia optional install.InstallUsing pip:pipinstalldatanonymizerUsing mimesis instead of the default Faker:pipinstalldatanonymizer[mimesis]Or from source:gitclonehttps://github.com/fgmacedo/datanonymizercddatanonymizerpythonsetup.pyinstallUsagePass your data throughstdinand get it back anonymized onstdout.NoteIn this case, the output will be equal to the input as no conversions were applied.catinput_file.csv|datanonymizer>output_file.csvUsing a config file to declare conversions and generators for the required fields:catinput_file.csv|datanonymizer--config./dataset_anon_config.yml>output_file.csvPlease see examples folder for a small demo:catexamples/small.csv|python-mdatanonymizer-i--configexamples/small_faker.yml--seedmy_seed>examples/small_anonymized_using_faker.csvOptional arguments:-h, --help            show this help message and exit
-l LANGUAGE, --language LANGUAGE
                      Language used by the Generator
-di DELIMITER_INPUT, --delimiter_input DELIMITER_INPUT
                      CSV delimiter
-do DELIMITER_OUTPUT, --delimiter_output DELIMITER_OUTPUT
                      CSV delimiter
-i, --ignore_errors   Continue on errors
--head HEAD           Outputs only the first <HEAD> lines
-g {faker,mimesis}, --generator {faker,mimesis}
                      Generator library to be used for fake data
--seed SEED           Seed for the pseudo random generator providers
--config CONFIG       Configuration fileConfig fileYou’l need a configuration file to setup transformations for each dataset.This file is a simpleyamlwhere you can configure fields.Field names should match the column name declared into the CSV input file.---fields:Task ID:omit:trueLocation:conversions:-fn:coords_to_h3kwargs:resolution:8Client Address:conversions:-fn:has_valuerename:has_addressCompany Name:generator:provider:business.companyrename:companyInvoice ID:generator:provider:person.identifierkwargs:mask:"#######"rename:invoiceGeneratorsThe generatos clause depends of the library you choose to provide fake data.You can use any generator available at the generic API from Faker or mimesis.For example, if you wanna mimic data with company names:Faker---fields:Company Name:generator:provider:companyMimesis---fields:Company Name:generator:provider:business.companyBut you can replace the real names by names of fruits (using Mimesis) or any other provider:---fields:Company Name:generator:provider:food.fruitOr generate random integers to replace real IDs:Faker---fields:ID:generator:provider:pyintkwargs:min_value:1max_value:15_000_000Mimesis---fields:ID:generator:provider:person.identifierkwargs:mask:"#######"ConversionsYou can apply any pre-configured conversion functions available.coords_to_h3has_value
Package: datanozzle
This library acts as a wrapper around the wonderful
[requests](https://python-requests.org) library for accessing
[Datagrepper](https://apps.fedoraproject.org/datagrepper/).RequirementsPython 3.3+[requests](https://python-requests.org)UsageThe API is designed around being chainable.  To start, create a newGrepper:>>> from datanozzle import Grepper
>>> g = Grepper()
>>>Then, you can build up queries by chaining methods (each new method returns a new
query object, so you can reuse partially built queries):>>> q1 = g.by_user('sross').ascending
>>> q2 = q1.by_topic('org.fedoraproject.prod.fedbadges.person.rank.advance')
>>>Queries are not executed until they are evaluated by iterating over them (the library will
automatically continue to fetch new pages of information).  Each record is returned as anEntry,
which has several persistent fields (which are accessible as properties), and additional data
available in a dict-like manner:>>> entry = next(iter(q2.take(pages=1)))
>>> entry.topic
'org.fedoraproject.prod.fedbadges.person.rank.advance'
>>> entry.timestamp
datetime.datetime(2015, 1, 16, 15, 2, 5)
>>> entry['person']['nickname']
'sross'
>>>Most queries can support multiple values.  You can either pass multiple values, likeg.by_user(‘sross’, ‘someuser’)or chain, likeg.by_user(‘sross’).by_user(‘someuser’).
To get a query with a parameter reset, callreset(parameter_name), likeg.reset(‘user’).Queriesby_user(*users): filter by FAS usernamewithout_user(*users): inverse ofby_userby_package(*packages): filter by package namewithout_package(*packages): inverse ofby_packageby_category(*categories): filter by category (the third or fourth part of the topic)without_category(*categories): inverse ofby_categoryby_topic(*topics): filter by topicwithout_topic(*topics): inverse ofby_topiccontaining(substring): filter by keyword in the messagepaginate(rows): set the number of rows per pagewith_meta(*meta_type): return additional meta-information with the resultsgrouped(property): group similar results togetherascending(property): sort results in ascending orderdescending(property): sort results in descending ordertake(pages): only return results from the firstpagespagesskip(pages): skip the firstpagespagesstarting_at(start): return results from after the given date/timeending_at(end): return results from before the given date/timedelta_seconds(delta): modify start/end times bydeltasecondsData Propertiescertificate: a string containing the certificatesignature: a string containing the signaturemeta: a dict containing additional meta-informationindex: the message indextimestamp: adatetimecontaining the timestamp for the objecttopic: the topic of the message
Package: data-nut-squirrel
data_squirrelWhat can finally be called a Production Remote Data Type Shared Python Framework
its basically pythonic pointersIt has the ability now to import classes as external importsRead this doc:Data Squirrel Documentationhttps://docs.google.com/document/d/1FRSHdIqurGtHC8vD-rI5Ryw-NAj9xbYtGOq5UFW55iU/editWatch these videos for explanation on use:How to build data squirrel shared python config filehttps://youtu.be/nLAUh-8oiTQHow to use data squirrel shared python config filehttps://youtu.be/t5BpfnzjGYo
Package: data-object
Description:Simple base class for creating data object (it means objects, dedicated
to storing data only). It’s a little bit similar tocase
classfrom
Scala. Using it as base class, You don’t need to take care of data
objects boilerplate (like __str__, __eq__, __hash__
methods). Main features: * Objects are considered to be equal based on
public fields (method with names started with _ are not took into
account) values instead of object identity * __str__ and
__repr__ methods return values based on public fields * Static
method for creating instances from dictionary. Main difference to dict
unpacking is that all non matching keys are ignored and default values
are supported * there are two base classes:DataObjectandImmutableDataObject. The second on creates immutable object (so
value assigned once to field, cannot be changed)WARNING:> In future version default behavior may change. It’s still
under consideration, whether two different classes with the same set of
fields and values should be equalInstallpip install data_objectUsageDataObjectfromdata_objectimportDataObjectclassCustomClass(DataObject):class_member='XYZ'def__init__(self,foo,bar):self.foo=fooself.bar=bar@propertydefother(self):return'ABC'Str and Repr:a=CustomClass('a','b')print(a)# output: CustomClass: {"bar": b, "class_member": XYZ, "foo": a, "other": ABC}print(repr(a))# output: CustomClass(bar=b, class_member=XYZ, foo=a, other=ABC)Equality and HashclassOtherClass(DataObject):def__init__(self,foo,bar):self.foo=fooself.bar=bara=CustomClass('a','b')b=CustomClass('a','b')c=CustomClass('a','z')d=OtherClass('a','b')a==b# Truea==c# Falsea==d# Truez=set()z.add(a)z.add(b)z.add(c)z.add(d)print(z)# output: {CustomClass(bar=z, foo=a), CustomClass(bar=b, foo=a)}Creating instances from dictclassOtherClass(DataObject):def__init__(self,a,b,c='xyz'):self.a=aself.b=bself.c=cparams1={'a':'aaa','b':'bbb','c':'ccc'}params2={'a':'xxx','b':'yyy'}params3={'a':'xxx'}inst1=OtherClass.from_dict(params1)inst2=OtherClass.from_dict(params2)inst3=OtherClass.from_dict(params3,none_if_not_found=True)print(inst1)# output: OtherClass: {"a": aaa, "b": bbb, "c": ccc}print(inst2)# output: OtherClass: {"a": xxx, "b": yyy, "c": xyz}print(inst3)# output: OtherClass: {"a": xxx, "b": None, "c": xyz}Copy instance (especially useful for immutable objects)primary_inst=CustomClass(foo='abc',bar='xyz')inst1=primary_inst.copy()inst2=primary_inst.copy(bar='aaa')inst3=primary_inst.copy(bar='aaa',other='some')print(repr(primary_inst))# output: CustomClass(bar=xyz, foo=abc)print(repr(inst1))# output: CustomClass(bar=xyz, foo=abc)print(repr(inst2))# output: CustomClass(bar=aaa, foo=abc)print(repr(inst3))# output: CustomClass(bar=aaa, foo=abc)ImmutableDataObjectfromdata_objectimportImmutableDataObjectclassCustomImmutableClass(ImmutableDataObject):def__init__(self,foo,bar):self.foo=fooself.bar=barinst=CustomImmutableClass('abc','xyz')print(inst)# output: CustomImmutableClass: {"bar": xyz, "foo": abc}inst.foo='aaa'will produce exception:Traceback (most recent call last):
...
data_object.exceptions.ImmutableObjectViolation: Changing attributes not permitted for immutable object
Package: dataopen-sdk-python
openapi 使用说明Client 参数说明字段类型默认值含义app_idstr无应用的唯一标识符app_secretstr无用于应用的安全认证的密钥urlstr or NoneType"https://analytics.volcengineapi.com"服务器的 URL 地址expirationstr or NoneType"1800"过期时间，单位是秒client.request 参数说明字段类型默认值含义service_urlstr无请求的服务 URL 地址methodstr无请求的 HTTP 方法，例如 "GET", "POST" 等headersDict[str, str]{}请求头，包含的信息如认证凭据，内容类型等paramsDict[str, any]{}URL 参数，用于 GET 请求bodyDict[str, any]{}请求体，通常在 POST 或 PUT 请求中包含发送的数据样例1、Get 方法fromdataopenimportClient,ParamsValueTypeapp_id=""app_secret=""dataopen_client=Client(app_id,app_secret)headers={}params:dict[str,ParamsValueType]={"app":46,"page_size":1,"page":1,}body={}res=dataopen_client.request("/dataopen/open-apis/xxx/openapi/v1/open/flight-list","GET",headers,params,body)print("\n\nOutput: ",res)2、Post 方法fromdataopenimportClient,ParamsValueTypeapp_id=""app_secret=""dataopen_client=Client(app_id,app_secret)headers={}params={}body:dict[str,any]={"uid_list":["1111111110000"],}res=dataopen_client.request("/dataopen/open-apis/xxx/openapi/v1/open/flight/version/6290880/add-test-user","POST",headers,params,body,)# Output resultsprint("\n\nOutput: ",res)
Package: dataos
quiet.pyPython ctypes bindings for libquiet to transmit data with sound.RequirementsnumpyInstallFor ARM platform, binary package is available on pypi, just usepipto install it:sudo apt install python-numpy
pip install --no-deps quiet.pyWe installnumpyseparately, as installingnumpyvia pip requires compiling numpy from source.For x86/amd64sudo apt install cmake
git clone https://github.com/xiongyihui/quiet.py && cd quiet.py
./scripts/libs.sh
pip install .UsageEncode a message, and then decode itfrom quiet import Encode, Decoder

def test():
    encoder = Encoder()
    decoder = Decoder()

    for chunk in encoder.encode('hello, world'):
        message = decoder.decode(chunk)
        if message is not None:
            print(message)


test()decode messages from recording in realtimeimport sys
import numpy
import pyaudio
from quiet import Encode, Decoder

def decode():
    if sys.version_info[0] < 3:
        import Queue as queue
    else:
        import queue

    FORMAT = pyaudio.paFloat32
    CHANNELS = 1
    RATE = 44100
    CHUNK = 16384  # int(RATE / 100)

    p = pyaudio.PyAudio()
    q = queue.Queue()

    def callback(in_data, frame_count, time_info, status):
        q.put(in_data)
        return (None, pyaudio.paContinue)

    stream = p.open(format=FORMAT,
                    channels=CHANNELS,
                    rate=RATE,
                    input=True,
                    frames_per_buffer=CHUNK,
                    stream_callback=callback)

    count = 0
    with Decoder(profile_name='ultrasonic-experimental') as decoder:
        while True:
            try:
                audio = q.get()
                audio = numpy.fromstring(audio, dtype='float32')
                # audio = audio[::CHANNELS]
                code = decoder.decode(audio)
                if code is not None:
                    count += 1
                    print(code.tostring().decode('utf-8', 'ignore'))
            except KeyboardInterrupt:
                break


decode()
Package: dataos-sdk-py
DataOS® offers a comprehensive range of powerful primitives that empower users to create, enhance, and seamlessly integrate their current capabilities. These versatile primitives serve as the building blocks for a wide array of functionalities.Designed to provide a robust foundation, the DataOS® software development kit (SDK) presents a cohesive set of APIs. These APIs enable smooth and intuitive interactions with the diverse range of primitives through their respective services. By utilizing the SDK, developers gain streamlined access to these building blocks, unlocking the potential to construct innovative solutions and integrate them seamlessly into their existing resources.Whether it's enhancing data management, refining analytical processes, or augmenting automation frameworks, the DataOS® SDK empowers developers to harness the full potential of DataOS®'s powerful primitives. This integration capability fosters a flexible and dynamic environment where users can tailor solutions to their unique needs and rapidly innovate with confidence.SDK CandidatesAsset Tags ServicecommonsDepot-ServiceGatewayHeimdallPorosStack Service
Package: data-overview
数据概览工具当前支持的功能：基于Pandas的DataFrame，生成该DataFrame的表和字段粒度概览，HTML格式。当前功能：数据表概览数据表行数，列数，内存大小各类型数据字段数量数据字段概览每个字段的类型，数量，去重后数量，缺失值数量，缺失率等数据字段详情每个字段的详细信息在研功能：导出excel和csv其他使用示例：参考data_overview/Usage.ipynb文件importpandasaspdimportdata_overviewasdo# 前提是先要执行：pip install data_overview进行安装# 载入数据df=pd.read_csv("./data/test_data.csv",encoding='gb18030')# 用法：输入df，可导出数据概览dfo=do.DataOverview(df)dfo.to_html("./data/report.html")# 导出路径和命名自定义
Package: datapackage-convert
Datapackage ConvertFull DocumentationConversions from tabular-data-packages. Currently:Merge mulitple datapackages into one.To SQLiteTo ParquetTo XLSXTo PostgresAll conversions aim to be memory efficiant and as fast they can be. This is the python library, providing bindings to therust library.Installpip install datapackage-convertUsageWhen refering to a datapackage you can either reference:Adatapackage.jsonfile.A datapackage directory containing adatapackage.jsonfile. e.g./a/datapackage/dirA zip file containing adatapackage.jsonfile. e.g.my_datapackage.zipMerge datapackageThis will merge datapckages into a single one.from datapackage_convert import merge_datapackage

merge_datapackage('output_datapackage', ['datapackage1', 'datapackage2'])To SQLiteWill convert to sqlite file. Adding foreign key relationships and indexes for them.from datapackage_convert import datapackage_to_sqlite

datapackage_to_sqlite('mydatabase.db', 'my_datapackage')To parquetWill convert to snappy compressed parquet files.from datapackage_convert import datapackage_to_parquet

datapackage_to_parquet('output_directory', 'my_datapackage')To xlsxWill convert to XLSX files.from datapackage_convert import datapackage_to_xlsx

datapackage_to_xlsx('output.xlsx', 'my_datapackage')To postgresWill insert data into postgres database.from datapackage_convert import datapackage_to_postgres

datapackage_to_xlsx('postgres://name:pass@host/dbname', 'my_datapackage')