Saravanan666/pypi-org huggingface dataset:
Package: dfcompy
dfcompyDescriptiondfcompyis a Python package that provides a comprehensive tool for comparing two Pandas DataFrame objects. It can identify rows that are inserted, deleted, or updated between two DataFrames, catering especially to data analysis and data cleaning processes.InstallationInstalldfcompyusing pip:pipinstalldfcompyUsageimportpandasaspdfromdfcompyimportDataFrameComparator# Create example DataFrames# ... [example DataFrame creation]# Create a DataFrameComparator instancecomparator=DataFrameComparator(df1,df2,on=['ID'],subset=['Name','Age'])# Detect deleted rowsprint("Deleted Rows:")print(comparator.rows_deleted())# Detect inserted rowsprint("\nInserted Rows:")print(comparator.rows_inserted())# Detect updated rowsprint("\nUpdated Rows:")print(comparator.rows_before_update())# Detect unchanged rowsprint("\nUnchanged Rows:")print(comparator.rows_in_common())ContributingContributions are welcome! For major changes, please open an issue first to discuss what you would like to change.LicenseMIT
Package: dfcon
dfcon : dataFileControllerTo make access to the database easier.InstallationpipinstalldfconRequirementsPython 3.xUsagemodule import ( and bref description )fromdfconimportDirectoryfromdfcon.path_filterimportDircFilter,FileFilterfromcmpfilterimportFilter,TiledFilter,OverlapedFilterFilterFilterFilteris the callable class that extends Python's conditional expressions.This can determine if an object meets the condition.Filteris abstruct class, the programmer can create filter classes for any object or data.OverlapedFilterOverlapedFilteris compound filter consisting of a Filter joined by the AND operator.This isFilter's subclass.filter1=MyFilter()filter2=MyFilter()...filters:OverlapedFilter=Filter.overlap([filter1,filter2,...])other method,filters=filter1&filter2TiledFilterTiledFilteris compound filter consisting of a Filter joined by the OR operator.This isFilter's subclass.filters:TiledFilter=Filter.tile([filter1,filter2,...])other method,filters=filter1|filter2DircFilterDircFiltermakes a judgment about the directory of the file path.This isFilter's subclass.dfilter=DircFilter().contained_path("abc")ifdfilter("./src/sample.py"):# False...ifdfilter("./abc/sample.py"):# True...DircFilterclass used inDirectoryand its some function's arguments.FileFilterFileFiltermakes a judgment about the filename of the file path.This isFilter's subclass.ffilter=(FileFilter().include_extention(["py","txt"]).exclude_extention(["c","cpp"]))ifffilter("./src/sample.py"):# True...ifffilter("./abc/sample.txt"):# True...ifffilter("./abc/sample.c"):# False...ifffilter("./abc/sample.cpp"):# False...FileFilterclass used inDirectoryand its some function's arguments.DirectoryDirectoryDirectoryinstance can be made bydirc=Directory(path="path/to/target")get_file_path(filters)filters, default toNone.Type offiltersisFilter | List[Filter] | NoneThis function return generator of file list which belongs topath.path="path/to/target"dirc=Directory(path)file_gen=dirc.get_file_path()You can applyfilters, such asFileFilterandDircFilter.
Package: dfcsv2parquet
converts large CSV files into smaller, Pandas-compatible Parquet filespip install dfcsv2parquetTested against Windows 10 / Python 3.10 / AnacondaThe convert2parquet function is used to convert large CSV files into smaller Parquet files.
It offers several advantages such as reducing memory usage,
improving processing speed, and optimizing data types for more efficient storage.This function might be interesting for individuals or organizations working with large datasets in CSV format
and looking for ways to optimize their data storage and processing.
By converting CSV files to Parquet format, which is a columnar storage format, several benefits can be achieved:Reduced Memory Usage:Parquet files are highly compressed and store data in a columnar format,
allowing for efficient memory utilization.
This can significantly reduce the memory footprint compared to traditional row-based CSV files.Improved Processing Speed:Parquet files are designed for parallel processing and can be read in a highly efficient manner.
By converting CSV files to Parquet, you can potentially achieve faster data ingestion and query performance.Optimized Data Types:The function includes a data type optimization step (optimize_dtypes) that aims to minimize the memory usage
of the resulting Parquet files. It intelligently selects appropriate data types based on the actual
data values, which can further enhance storage efficiency.Categorical Data Optimization:The function handles categorical columns efficiently by limiting the number of categories (categorylimit).
It uses the union_categoricals function to merge categorical data from different chunks,
reducing duplication and optimizing memory usage.Args:csv_file(str|None):PathtotheinputCSVfile.DefaultisNone.parquet_file(str|None):PathtotheoutputParquetfile.DefaultisNone.chunksize(int):NumberofrowstoreadfromtheCSVfileperchunk.Defaultis1000000.categorylimit(int):Theminimumnumberofcategoriesincategoricalcolumns.Defaultis4.verbose(int|bool):Verbositylevel.Setto1orTrueforverboseoutput,0orFalsefornooutput.Defaultis1.zerolen_is_na(int|bool):Whethertotreatzero-lengthstringsasNaNvalues.Setto1orTruetoenable,0orFalsetodisable.Defaultis0.args:passedtopd.read_csv(doesn't work with the cli)kwargs:passedtopd.read_csv(doesn't work with the cli)Returns:NoneExamples:# Download the csv:https://github.com/hansalemaos/csv2parquet/raw/main/bigcsv.part03.rarhttps://github.com/hansalemaos/csv2parquet/raw/main/bigcsv.part02.rarhttps://github.com/hansalemaos/csv2parquet/raw/main/bigcsv.part01.rar# in Pythonfromdfcsv2parquetimportconvert2parquetconvert2parquet(csv_file=r"C:\bigcsv.csv",parquet_file=r'c:\parquettest4.pqt',chunksize=1000000,categorylimit=4,verbose=True,zerolen_is_na=False,)# CLIpython.exe"...\__init__.py"--csv_file"C:\bigcsv.csv"--parquet_file"c:\parquettest4.pqt"--chunksize100000--categorylimit4--verbose1--zerolen_is_na1
Package: df-csv-excel
read_csv_excelOverviewread_csv_excelis a Python library that provides a collection of common and customized functions for handling data, particularly CSV and Excel data. It simplifies the process of converting data into Pandas DataFrames, performing statistical analysis, and creating visualizations.DocumentDocumentFeaturesCSV and Excel data conversion to Pandas DataFrames.Statistical analysis of DataFrames.Customized functions for common data handling tasks.Plotting functions for data visualization.InstallationYou can installread_csv_excelusing pip:pip install df-csv-excelpypi page link is here:(https://pypi.org/project/df-csv-excel/)Usage read and process dataHere is some examples for the functions, for more details, please refer to (https://pypi.org/project/df-csv-excel/)from df_csv_excel import read_data 

df = read_data.read_data_by_path('a.xlsx')
df['name'] = read_data.get_feature_from_json(df, 'json_column', ['key_name1', 'key_name2', 'key_name3])
df['time_column'] = read_data.foramt_date_column(df, 'date')Usage plot datafrom df_csv_excel import read_data, plot_data 

df = read_data.read_data_by_path('a.xlsx')
plot_data.plot_histgram(df, 'column_name')ContributingIf you have suggestions, enhancements, or find issues, please feel free to contribute! Follow the Contribution Guidelines for more details.
Package: dfdata
dfdatadfdata 用于下载金融数据。将数据源的数据保存到本地，默认sqlite数据库。安装pip install dfdata使用下载tushare pro 数据源为例。使用pip install tushare安装好tushare，再设置一次tushare的token，它会保存在用户目录下tk.csv。import tushare as ts
ts.set_token('your_tushare_token')接下载可以使用dfdata下载和读取数据。import dfdata as dd

# 下载期货日线行情
dd.save_futures_daily(source='tushare')

# 读取期货日线行情
dd.read_futures_daily(source='tushare')更多使用方法见文档：http://dfdata.org/
Package: dfdb
dfdbPandas-like interface for manipulating SQL databasesInstallpip install dfdbOverviewThis package is a lightweight wrapper around SQLalchemy. As such, it can be used with a variety of database backends, although as of now the package has only been tested with SQLite.Thedfdb.Databaseclass provides a dict-like interface of manipulating tables in a SQL database. For example, if we create aDatabaseobject likedb = dfdb.Database(filename='data.db', drivername='sqlite')Then we can perform operations on a table calleddfindata.dbby calling, for exampledb['df'].groupby('col1').agg(mean=('col2', 'mean')).fetch()db['df']is adfdb.DataFrameobject that behaves almost identically as apandas.DataFrame. The only difference is that you need to call.fetch(), which establishes a connection to the database and performs the corresponding SQL query.Adding rows to the table is as simple asdb['df'].append({'col1': 1, 'col2': 2})More detailed documentation to follow in the future.
Package: df-db-connector
Dialog Flow DB ConnectorThereis an addon for theDialog Flow Framework, a minimalistic open-source engine for conversational services.Dialog Flow DB Connectorallows you to to save and retrieve user dialogue states (in the form of aContextobject) using various database backends.Currently, the supported options are:jsonpickleshelveSqlitePostgresqlMySQLMongoDBRedisYDBAside from this, we offer some interfaces for saving data to your local file system. These are not meant to be used in production, but can be helpful for prototyping your application.Quick StartInstallationpipinstalldf-db-connectorPlease, note that if you are going to use one of the database backends, you will have to specify an extra or install the corresponding requirements yourself.pipinstalldf-db-connector[redis]pipinstalldf-db-connector[mongodb]pipinstalldf-db-connector[mysql]pipinstalldf-db-connector[postgresql]pipinstalldf-db-connector[sqlite]pipinstalldf-db-connector[ydb]Basic examplefromdf_engine.coreimportContext,Actorfromdf_db_connectorimportSQLConnectorfrom.scriptimportsome_df_scriptdb=SQLConnector("postgresql://user:password@host:port/dbname")actor=Actor(some_df_script,start_label=("root","start"),fallback_label=("root","fallback"))defhandle_request(request):user_id=request.args["user_id"]ifuser_idnotindb:context=Context(id=user_id)else:context=db[user_id]new_context=actor(context)db[user_id]=new_contextassertuser_idindbreturnnew_context.last_responseTo get more advanced examples, take a look atexampleson GitHub.Contributing to the Dialog Flow DB ConnectorPlease refer toCONTRIBUTING.md.
Package: df-diskcache
df-diskcacheSummaryInstallationFeaturesUsageSet TTL for cache entriesDependenciesSummarydf-diskcacheis a Python library for cachingpandas.DataFrameobjects to local disk.Installationpip install df-diskcacheFeaturesSupports the following methods:get: Get a cache entry (pandas.DataFrame) for the key. ReturnsNoneif the key is not found.set: Create a cache entry with an optional time-to-live (TTL) for the key-value pair.updatetouch: Update the last accessed time of a cache entry to extend the TTL.deleteprune: Delete expired cache entries.Dictionary-like operations:__getitem____setitem____contains____delitem__UsageSample Code:importpandasaspdfromdfdiskcacheimportDataFrameDiskCachecache=DataFrameDiskCache()url="https://raw.githubusercontent.com/pandas-dev/pandas/v2.1.3/pandas/tests/io/data/csv/iris.csv"df=cache.get(url)ifdfisNone:print("cache miss")df=pd.read_csv(url)cache.set(url,df)else:print("cache hit")print(df)You can also use operations like a dictionary:Sample Code:importpandasaspdfromdfdiskcacheimportDataFrameDiskCachecache=DataFrameDiskCache()url="https://raw.githubusercontent.com/pandas-dev/pandas/v2.1.3/pandas/tests/io/data/csv/iris.csv"df=cache[url]ifdfisNone:print("cache miss")df=pd.read_csv(url)cache[url]=dfelse:print("cache hit")print(df)Set TTL for cache entriesSample Code:importpandasaspdfromdfdiskcacheimportDataFrameDiskCacheDataFrameDiskCache.DEFAULT_TTL=10# you can override the default TTL (default: 3600 seconds)cache=DataFrameDiskCache()url="https://raw.githubusercontent.com/pandas-dev/pandas/v2.1.3/pandas/tests/io/data/csv/iris.csv"df=cache.get(url)ifdfisNone:df=pd.read_csv(url)cache.set(url,df,ttl=60)# you can set a TTL for the key-value pairprint(df)DependenciesPython 3.7+Python package dependencies (automatically installed)
Package: df-engine
Dialog Flow EngineThe Dialog Flow Engine (DFE) allows you to write conversational services. The service is written by defining a special dialog graph that describes the behavior of the dialog service. The dialog graph contains the dialog script. DFE offers a specialized language (DSL) for quickly writing dialog graphs. You can use it in such services for writing skills for Amazon Alexa and etc, chat-bots for social networks, websites call-centers and etc.Quick StartInstallationpipinstalldf_engineBasic examplefromdf_engine.core.keywordsimportGLOBAL,TRANSITIONS,RESPONSEfromdf_engine.coreimportContext,Actorimportdf_engine.conditionsascndfromtypingimportUnion# create script of dialogscript={GLOBAL:{TRANSITIONS:{("flow","node_hi"):cnd.exact_match("Hi"),("flow","node_ok"):cnd.true()}},"flow":{"node_hi":{RESPONSE:"Hi!!!"},"node_ok":{RESPONSE:"Okey"},},}# init actoractor=Actor(script,start_label=("flow","node_hi"))# handler requestsdefturn_handler(in_request:str,ctx:Union[Context,dict],actor:Actor):# Context.cast - gets an object type of [Context, str, dict] returns an object type of Contextctx=Context.cast(ctx)# Add in current context a next request of userctx.add_request(in_request)# pass the context into actor and it returns updated context with actor responsectx=actor(ctx)# get last actor response from the contextout_response=ctx.last_response# the next condition branching needs for testingreturnout_response,ctxctx={}whileTrue:in_request=input("type your answer: ")out_response,ctx=turn_handler(in_request,ctx,actor)print(out_response)When you run this code, you get similar output:type your answer: hi
Okey
type your answer: Hi
Hi!!!
type your answer: ok
Okey
type your answer: ok
OkeyTo get more advanced examples, take a look atexampleson GitHub.Contributing to the Dialog Flow EnginePlease refer toCONTRIBUTING.md.
Package: dfetch
DFetch can manage dependenciesWe needed a dependency manager that was flexible enough to retrieve dependencies as plain text
from various sources.svn externals,git submodulesandgit subtreessolve a similar
problem, but not in a VCS agnostic way or completely user friendly way.
We want self-contained code repositories without any hassle for end-users.
Dfetch must promote upstreaming changes, but allow for local customizations.
The problem is described thoroughlyhereand sometimes
is also known asvendoring.Other tools that do similar things areZephyr's West,CMake ExternalProjectand other meta tools.
Seealternativesfor a complete list.Getting started|Manual|Troubleshooting|ContributingInstallStablepipinstalldfetchlatest versionpipinstallhttps://github.com/dfetch-org/dfetch/archive/main.zip
Package: df-extract
DF Extract LibRequirementsPython 3.10+ asyncioInstallation# Using pip$python-mpipinstalldf-extract# Manual install$python-mpipinstall.1. To extract content fromPDFfromdf_extract.pdfimportExtractPDFpath="/home/test/ABC.pdf"extract_pdf=ExtractPDF(file_path=path)# By default, output as textawaitextract_pdf.extract()# Output will be located `/home/test/ABC.pdf.txt`# Output as jsonawaitextract_pdf.extract(as_json=True)# Output will be located `/home/test/ABC.pdf.json`You can change the output directory with simply passoutput_dirparamfromdf_extract.pdfimportExtractPDFpath="/home/test/ABC.pdf"extract_pdf=ExtractPDF(file_path=path,output_dir="/home/test/output")awaitextract_pdf.extract()Extract content fromPDFwith image dataThis requireseasyocrfromdf_extract.baseimportImageExtractfromdf_extract.pdfimportExtractPDFpath="/home/test/ABC.pdf"image_extract=ImageExtract(model_download_enabled=True)extract_pdf=ExtractPDF(file_path=path,image_extract=image_extract)awaitextract_pdf.extract()2. To extract content fromPPTandPPTxfromdf_extract.pptximportExtractPPTxpath="/home/test/DEF.pptx"extract_pptx=ExtractPPTx(file_path=path)# By default, output as textawaitextract_pptx.extract()# Output will be located `/home/test/DEF.pptx.txt`# Output as jsonawaitextract_pptx.extract(as_json=True)# Output will be located `/home/test/DEF.pptx.json`3. To extract content fromDocandDocxfromdf_extract.docximportExtractDocxpath="/home/test/GHI.docx"extract_docx=ExtractDocx(file_path=path)# By default, output as textawaitextract_docx.extract()# Output will be located `/home/test/GHI.docx.txt`# Output as jsonawaitextract_docx.extract(as_json=True)# Output will be located `/home/test/GHI.docx.json`4. To extract content fromPNG,JPEGandJPGfromdf_extract.imageimportExtractImagepath="/home/test/JKL.png"extract_png=ExtractImage(file_path=path)# By default, output as textawaitextract_png.extract()# Output will be located `/home/test/JKL.png.txt`# Output as jsonawaitextract_png.extract(as_json=True)# Output will be located `/home/test/JKL.png.json`
Package: dffml
Mission StatementDFFML aims to be the easiest and most convenient way to use Machine Learning.Its a machine learning distribution. Providing you access to a set of popular
machine learning libraries guaranteed to work together.Its a AI/ML Python library, command line application, and HTTP service.You give it your data and tell it what kind of model you want to train. It
creates a model for you.If you want finer grained control over the model, you can easily do so by
implementing your own model plugin.We make it easy to use and deploy your models.We provide a directed graph concurrent execution environment with managed
locking which we call DataFlows.DataFlows make it easy to generate datasets or modify existing datasets for
rapid iteration on feature engineering.DocumentationDocumentation for the latest release is hosted athttps://intel.github.io/dffml/Documentation for the master branch is hosted athttps://intel.github.io/dffml/master/index.htmlContributingThecontributing pagewill guide you through getting setup and contributing to DFFML.HelpAsk a question via anissueSend an email todffml-users@lists.01.orgYou can subscribe to the users mailing list herehttps://lists.01.org/postorius/lists/dffml-users.lists.01.org/Ask a question on theGitter chatLicenseDFFML is distributed under theMIT License.LegalThis software is subject to the U.S. Export Administration Regulations and
other U.S. law, and may not be exported or re-exported to certain countries
(Cuba, Iran, Crimea Region of Ukraine, North Korea, Sudan, and Syria) or to
persons or entities prohibited from receiving U.S. exports (including
Denied Parties, Specially Designated Nationals, and entities on the Bureau
of Export Administration Entity List or involved with missile technology or
nuclear, chemical or biological weapons).
Package: dffml-feature-auth
DFFML auth Featuresauth description.UsageExample usage$dffmloperationsrepo\-sourcesmemory=nill\-logdebug\-keysfeedface\-repo-defUnhashedPassword\-opsscryptget_single\-opimpn-memory-opimpsscryptget_single\-remapget_single.ScryptPassword=scrypt\-output-specs'["ScryptPassword"]=get_single_spec'[{"extra": {},"features": {"scrypt": {"hashed":"322e95513218f1b70c34cc8af4eafdc18c44946c51b5b7cfd4c0258047d38942dd153822ad118fe908cfa15a1b315402772ea33608bcaf20f6a5e32dc4891a50","salt":"e78f23e1a05165211d50652e7a07e769ade1192c126d049b022a10d5b8ff1a3ff474666824915c9e318b2f06a37d52ff1dfcc8e9f56873d0b083ee2d43f7b9c7"}},"last_updated": "2019-05-10T15:58:19Z","key": "feedface"}]LicenseDFFML DFFML auth are distributed under theMIT License.
Package: dffml-feature-git
DFFML Features For Git Version ControlGit features scrape data from Git repositories.DemoUsageScrape lines of comments to lines of code ratio, diversity of authorship, number
of commits, number of authors, and frequency of release data for a given Git
repo.export OPIMPS="group_by quarters_back_to_date check_if_valid_git_repository_URL clone_git_repo git_repo_default_branch git_repo_checkout git_repo_commit_from_date git_repo_author_lines_for_dates work git_repo_release lines_of_code_by_language lines_of_code_to_comments git_commits count_authors cleanup_git_repo"dffml operations repo \-log debug \-keys https://github.com/intel/dffml \-repo-def URL \-remap \group_by.cloc=cloc \group_by.release=release \group_by.work=work \group_by.commits=commits \group_by.authors=authors \group_by.relase=release \-dff-memory-operation-network-ops $OPIMPS \-dff-memory-opimp-network-opimps $OPIMPS \-inputs \{0,1,2,3,4,5,6,7,8,9}=quarter \"'2019-03-29 13:24'=quarter_start_date" \True=no_git_branch_given \-output-specs '{"cloc": {"group": "quarter","by": "language_to_comment_ratio","fill": 0},"authors": {"group": "quarter","by": "author_count","fill": 0},"work": {"group": "quarter","by": "work_spread","fill": 0},"release": {"group": "quarter","by": "release_within_period","fill": False},"commits": {"group": "quarter","by": "commit_count","fill": 0}}=group_by_spec'TODOTransformsTake data of one defintion and label it as another definition.{"defintions":{},"operations":{},"transforms":{"quarter_date_to_git_date":{"quarter_date":["git_date"]},"thing_to_other_data_types":{"thing":["first_data_type","second_data_type"]},},}LicenseDFFML DFFML Features For Git Version Control are distributed under theMIT License.
Package: dffml-model-pytorch
DFFML PyTorch ModelsAboutdffml_model_pytorch supports pre-trained PyTorch ConvNet models.DocumentationDocumentation is hosted athttps://intel.github.io/dffml/plugins/dffml_model.html#dffml-model-pytorchLicensedffml_model_pytorch Models are distributed under the terms of theMIT License.
Package: dffml-model-scikit
DFFML Models For scikit / sklearnAboutModels created using scikit.Install$python3-mpipinstall--userdffml-model-scikitUsageLinear Regression ModelFor implementing linear regression to a dataset, let us take a simple example:Years of ExperienceExpertiseTrust FactorSalary0010.2101030.4202050.6303070.8404091.0505111.260$cat>train.csv<<EOFYears,Expertise,Trust,Salary0,1,0.2,101,3,0.4,202,5,0.6,303,7,0.8,40EOF$cat>test.csv<<EOFYears,Expertise,Trust,Salary4,9,1.0,505,11,1.2,60EOF$dffmltrain\-modelscikitlr\-model-featuresYears:int:1Expertise:int:1Trust:float:1\-model-predictSalary\-model-directorytempdir\-sourcesf=csv\-source-filenametrain.csv\-source-readonly\-logdebug$dffmlaccuracy\-modelscikitlr\-model-featuresYears:int:1Expertise:int:1Trust:float:1\-model-predictSalary\-model-directorytempdir\-sourcesf=csv\-source-filenametest.csv\-source-readonly\-logdebug$echo-e'Years,Expertise,Trust\n6,13,1.4\n'|\dffmlpredictall\-modelscikitlr\-model-featuresYears:int:1Expertise:int:1Trust:float:1\-model-predictSalary\-model-directorytempdir\-sourcesf=csv\-source-filename/dev/stdin\-source-readonly\-logdebugLicenseScikit Models are distributed under the terms of theMIT License.
Package: dffml-model-scratch
DFFML Scratch ModelsAboutModels created without a machine learning framework.Install$python3-mpipinstall--userdffml-model-scratchUsageIf we have a dataset of years of experience in a job and the Salary (in
thousands) at that job we can use the Simple Linear Regression model to predict
a salary given the years of experience (or the other way around).First we create the file containing the dataset. Then we train the model, get
its accuracy. And usingechopipe a new csv file of data to predict into the
model, and it will give us it prediction of the Salary.$cat>dataset.csv<<EOFYears,Salary1,402,503,604,705,80EOF$dffmltrain-modelscratchslr-model-featuresYears:int:1-model-predictSalary-model-directorytempdir-sourcesf=csv-source-filenamedataset.csv-source-readonly-logdebug$dffmlaccuracy-modelscratchslr-model-featuresYears:int:1-model-predictSalary-model-directorytempdir-sourcesf=csv-source-filenamedataset.csv-source-readonly-logdebug1.0$echo-e'Years,Salary\n6,0\n'|dffmlpredictall-modelscratchslr-model-featuresYears:int:1-model-predictSalary-model-directorytempdir-sourcesf=csv-source-filename/dev/stdin-source-readonly-logdebug[{"extra": {},"features": {"Salary": 0,"Years": 6},"last_updated": "2019-07-19T09:46:45Z","prediction": {"confidence": 1.0,"value": 90.0},"key": "0"}]LicenseScratch Models are distributed under the terms of theMIT License.