Saravanan666/pypi-org huggingface dataset:
Package: pandasnet-netstandard
pandasnetpandasnet_netstandardis a python package build on top ofpythonnet_netstandard.
It provides additional data conversions forpandas,numpyanddatetimePrerequisitespython 3.6 or higher.dotnet.dotnet also providesscriptsto proceed the installation by command line.Installationpip install pandasnet_netstandardFeaturesTo load the converter you need to import the package once in your python environment.
If the dotnet clr isn't started yet through the pytonnet package the import will.importpandasnet_netstandardWe construct a simple C# function to test conversionusingSystem;usingSystem.Collections.Generic;namespaceLibForTests{publicclassPandasNet{publicstaticDictionary<string,Array>BasicDataFrame(Dictionary<string,Array>df)=>df;}}We build this function into a library namedLibForTests.dll.
We load this library into our python environment then use it.importclrimportpandasnet_netstandard# Load the convertersimportpandasaspdfromdatetimeimportdatetime# Load your dllclr.AddReference('LibForTests.dll')fromLibForTestsimportPandasNetaspdnetx=pd.DataFrame({'A':[1,2,3],'B':[1.23,1.24,1.22],'C':['foo','bar','other'],'D':[datetime(2021,1,22),datetime(2021,1,23),datetime(2021,1,24)]})y=pdnet.BasicDataFrame(x)print(y)Below an exhausitve list of supported data convertions.Python -> .NetPython.Netdatetime.datetimeDateTimedatetime.dateDateTimedatetime.timedeltaTimeSpandatetime.timeTimeSpannumpy.ndarray(dtype=bool_)bool[]numpy.ndarray(dtype=int8)sbyte[]numpy.ndarray(dtype=int16)short[]numpy.ndarray(dtype=int32)int[]numpy.ndarray(dtype=int64)long[]numpy.ndarray(dtype=uint8)byte[]numpy.ndarray(dtype=uint16)ushort[]numpy.ndarray(dtype=uint32)uint[]numpy.ndarray(dtype=uint64)ulong[]numpy.ndarray(dtype=float32)float[]numpy.ndarray(dtype=float64)double[]numpy.ndarray(dtype=datetime64)DateTime[]numpy.ndarray(dtype=timedelta64)TimeSpan[]numpy.ndarray(dtype=str)string[]pandas._libs.tslibs.timestamps.TimestampDateTimepandas._libs.tslibs.timedeltas.TimeDeltaTimeSpanpandas.core.series.SeriesArraypandas.core.frame.DataFrameDictionary[string, Array].Net -> Python.NetPythonDateTimedatetime.datetimeTimeSpandatetime.timedeltabool[]numpy.ndarray(dtype=bool_)sbyte[]numpy.ndarray(dtype=int8)short[]numpy.ndarray(dtype=int16)int[]numpy.ndarray(dtype=int32)long[]numpy.ndarray(dtype=int64)byte[]numpy.ndarray(dtype=uint8)ushort[]numpy.ndarray(dtype=uint16)uint[]numpy.ndarray(dtype=uint32)ulong[]numpy.ndarray(dtype=uint64)float[]numpy.ndarray(dtype=float32)double[]numpy.ndarray(dtype=float64)DateTime[]numpy.ndarray(dtype=datetime64)TimeSpan[]numpy.ndarray(dtype=timedelta64)Dictionary[string, Array]pandas.core.frame.DataFrameContributingIssue tracker:https://github.com/fdieulle/pandasnet/issuesIf you want to checkout the project and propose your own contribution, you will need to setup it following few steps:Create a virtual environment:python -m venv venvActivate your virtual environment:venv/Scripts/activateInstall package dependenciespip install -r requirements.txtLicenseThis project is open source under theMIT license.
Package: pandas-nlp
Pandas NLPIt's an extension for pandas providing some NLP functionalities for strings.SetupRequirementspython >= 3.8InstallationExecute:pipinstall-Upandas-nlpTo install the default spacy English model:spacyinstallen_core_web_mdKey featuresLanguage detectionimportpandasaspdimportpandas_nlppandas_nlp.register()df=pd.DataFrame({"id":[1,2,3,4,5],"text":["I like cats","Me gustan los gatos","M'agraden els gats","J'aime les chats","Ich mag Katzen",],})df.text.nlp.language()Output0    en
1    es
2    ca
3    fr
4    de
Name: text_language, dtype: objectwith confidence:df.text.nlp.language(confidence=True).apply(pd.Series)Outputlanguage  confidence
0       en    0.897090
1       es    0.982045
2       ca    0.999806
3       fr    0.999713
4       de    0.997995String embeddingimportpandasaspdimportpandas_nlppandas_nlp.register()df=pd.DataFrame({"id":[1,2,3],"text":["cat","dog","violin"]})df.text.nlp.embedding()Output0    [3.7032, 4.1982, -5.0002, -11.322, 0.031702, -...
1    [1.233, 4.2963, -7.9738, -10.121, 1.8207, 1.40...
2    [-1.4708, -0.73871, 0.49911, -2.1762, 0.56754,...
Name: text_embedding, dtype: objectClosest conceptimportpandasaspdimportpandas_nlppandas_nlp.register()themed=pd.DataFrame({"id":[0,1,2,3],"text":["My computer is broken","I went to a piano concert","Chocolate is my favourite","Mozart played the piano"]})themed.text.nlp.closest(["music","informatics","food"])Output0    informatics
1          music
2           food
3          music
Name: text_closest, dtype: objectSentence extractionimportpandasaspdimportpandas_nlppandas_nlp.register()df=pd.DataFrame({"id":[0,1],"text":["Hello, how are you?","Code. Sleep. Eat"]})df.text.nlp.sentences()Output0[Hello,howareyou?]1[Code.,Sleep.,Eat]Name:text_sentences,dtype:object
Package: pandas-nosql
Pandas-NoSQLPandas-NoSQL adds read and write capabilities to pandas for several nosql databasesPandas read and write methods for:MongoDBElasticsearchRedisApache CassandraThe dependencies used for the functions are:pymongo by MongoDBelasticsearch by Elasticredis-py by Rediscassandra-driver by DataStaxInstallation$pipinstallpandas-nosqlDocumentationMongoDB# Read a MongoDB collection into a DataFrame# Defaults: host='localhost' port=27017# For nested collections use normalize=Trueimportpandasaspdimportpandas_nosqldf=pd.read_mongo(database='test_db',collection='test_col',normalize=True,host='localhost',port=27017)# Write DataFrame to MongoDB collection# modify_colletion parameter is to help prevent accidental overwritesdf.to_mongo(database='test_db',collection='test_col2',mode='a',host='localhost',port=27017)Elasticsearchimportpandasaspdimportpandas_nosql# Read an Elastic index into a DataFrame# To Access localhost:#   * hosts='https://localhost:9200'#   * verify_certs=False# If "xpack.security.enabled: false" use http instead# To split out _source use split_source=Trueelastic_cols=('make','model','purchase_date','miles')df=pd.read_elastic(hosts='https://localhost:9200',username='elastic',password='strong_password',index='test_index',fields=elastic_cols,verify_certs=Falsesplit_source=True)# Write DataFrame to Elastic Indexdf.to_elastic(hosts='https://localhost:9200',username='elastic',password='strong_password',index='test_index',mode='w')Redisimportpandasaspdimportpandas_nosql# Read a DataFrame that was sent to Redis using the to_redis method# A DataFrame not sent to Redis using the to_redis method is not guaranteed to be read properly# To Access localhost:#   * host='localhost'# To persist the DataFrame in Redis use expire_seconds=None# To set an expiration for the DataFrame pass an integer to expire_secondsdf=pd.read_redis(host='localhost',port=6379,redis_key='test_key')# Write a DataFrame to Redisdf.to_redis(host='localhost',port=6379,redis_key='test_key',expire_seconds=None)Apache Cassandraimportpandasaspdimportpandas_nosql# Read an Apache Cassandra table into a Panda DataFrame# To Access localhost:#   * contact_points=['localhost']#   * contact_points must be a listdf=pd.read_cassandra(contact_points=['localhost'],port=9042,keyspace='test_keyspace',table='test_table')# Append a DataFrame to Apache Cassandra# DataFrame Columns must match table Columnsreplication={'class':'SimpleStrategy','replication_factor':1}df.to_cassandra(contact_points=['localhost'],port=9042,keyspace='test_keyspace',table='test_table',mode='w',replication=replication)
Package: pandas-npi
pandas-npiNPI validation made easy with PandasInstallationpip install pandas-npiBasic UsageUpdate definitionsimport pandas_npi

pandas_npi.update_definitions()

#------------------------------additional details------------------------------

#CMS updates the NPPES dataset about once per month. Running the above code
#on a monthly basis will ensure that you are using the most up-to-date information.

#Note that the function above has no parameters. Just run the function, and it will
#notify you when the update is complete.Validate NPIimport pandas as pd
import pandas_npi

df = pd.read_csv("your_dataset_here.csv")

df = pandas_npi.validate(df, "npi_field_here")

#------------------------------additional details------------------------------

#The validate function takes care of cleaning/standardizing your npi. No need to
#preprocess on your end! Even special characters and letters are removed.

#Your dataset will be compared against the NPPES dataset. Four new fields will
#be added to your dataset: nppes_name, nppes_type, nppes_status, and nppes_deactivation_date.
#For full description of new fields, see below:nppes_name - gives the provider or facility name for the given NPInppes_type - shows whether the NPI is a provider or facility NPI. Be aware that sole proprietors
are not required to bill under a facility NPInppes_status - shows whether an NPI is active or deactivatednppes_deactivation_date - If the NPI has been deactivated, lists the deactivation date. Else is NaN.A final note - If an NPI does not exist now, and has never existed, nppes_name, nppes_type, and
nppes_status will all be labeled "invalid".
Package: pandas-ods-reader
pandas-ods-readerProvides a function to read in a.odsor.fodsfile and returns a pandas DataFrame.It usesezodfto read in.odsfiles. Since.fodsfiles are essentially xml,lxmlis used to read them. The correct parser is automatically chosen based on the file's extension.If a range is specified in the sheet to be imported, it seems thatezodfimports empty cells as well. Therefore, completely empty rows and columns are dropped from the DataFrame, before it is returned. Only trailing empty rows and columns are dropped.If the ODS file contains duplicated column names, they will be numbered and the number is appended to the column name in the resulting DataFrame.DependenciesezodflxmlpandasInstallationpip install pandas-ods-readerUsagefrompandas_ods_readerimportread_odspath="path/to/file.ods"# by default the first sheet is importeddf=read_ods(path)# load a sheet based on its index (1 based)sheet_idx=2df=read_ods(path,sheet_idx)# load a sheet based on its namesheet_name="sheet1"df=read_ods(path,sheet_name)# load a file that does not contain a header row# if no columns are provided, they will be numbereddf=read_ods(path,1,headers=False)# load a file and provide custom column names# if headers is True (the default), the header row will be overwrittendf=read_ods(path,1,columns=["A","B","C"])
Package: pandas-oop
Pandas-Oop(Also known as Poop), is a package that uses Pandas dataframes with object oriented programming styleInstallation:pipinstallpandas-oopSome examplesfrompandas_oopimportmodelsfrompandas_oop.fieldsimportStringColumn,IntegerColumn,FloatColumn,DateColumn,BoolColumnDB_CONNECTION=models.Connection('sqlite:///pandas_oop.db')# this is the same con_string for sqlalchemy engine@models.sql(table='people',con=DB_CONNECTION)# Use this decorator if you want to connect your class to a database@models.DataclassPeople(models.DataFrame):name=StringColumn(unique=True)age=IntegerColumn()money=FloatColumn(target_name="coins")# target_name if the name in the csv or table is coins and you want to have a different variable nameinsertion_date=DateColumn(format='%d-%m-%Y')is_staff=BoolColumn(true='yes',false='no')Now when instantiating this class, it will return a custom dataframe with all the functionalities of a Pandas
dataframe and some otherspeople=People()"""-----------------------------------------------------------"""people=People(from_csv=DATA_FILE,delimiter=";")"""-----------------------------------------------------------"""people=People(from_sql_query='select * from people')"""-----------------------------------------------------------"""people=People(from_df=some_dataframe)"""-----------------------------------------------------------"""people=People(from_iterator=some_function_that_yield_values)"""-----------------------------------------------------------"""forpeople_chunkinPeople(from_csv=DATA_FILE,delimiter=";",chunksize=10):...example of function that yield values:defsome_function_that_yield_values():whilesomething:...yieldname,age,money,insertion_date,is_staffYou can also save it to the database with the save() method (if the dtypes of the columns change, this will raise a
ValidationError):people.save()You can upsert to the database and this will automatically look at the unique fields that were declared in the classpeople.save(if_row_exists='update')orpeople.save(if_row_exists='ignore')If you want to revalidate your dataframe (convert the columns dtypes to the type that was declared in the class), you can
call the validate() method:people.validate()You can also validate from another class. For example, you can do something like this:people=People(from_csv=DATA_FILE)jobs=Jobs(from_sql_query='select * from jobs')people_with_jobs=people.merge(jobs,on='name').validate(from_class=PeopleWithJobs)This is the list of the overriten methods that return a pandas_oop custom dataframe'isnull''head''abs''merge''loc' and dataframe slicingI will add more and more methods on this list.New featuresAlembic Database migration support added:On your main application package, import Base (this is a declarative_base from sqlalchemy)frompandas_oopimportBaseAdd this configuration on the env.py file of your alembic configfromyour_appimportBasetarget_metadata=Base.metadataAnd finaly, update your database url on your alembic.ini file
Package: pandas-or
Pandas OROptimization research made easy usingPandasandOrtools.The purpose of working with Pandas in combination with Ortools is to bridge the gap between
optimization research and data analytics, data science, and data visualization.
With pandas-or, you can solve complex knapsack or vehicle routing type problems and then create
statistics, plots, and dashboards from your results!💻 InstallationPippip install pandas-or▶ DemoTry out the examples in Binder!🚶 Upcoming featuresCode tests.Basic shift scheduling solvers.More complex routing problem solvers (pickups & deliveries, constraints).Read The Docs documentation.Gitlab CI builds.📚 Read The DocsComing soon!Sponsors
Package: pandas-oracle
This package includes methods for:Running a specified query, passed in as a string, on an Oracle database and returning the result to a Pandas data frame.Executing a command with no return data.Inserting the contents of a Pandas data frame into an Oracle database table.Installationpip install pandas_oracleOverviewSample usage:import pandas_oracle.tools as pt

query1 = "select id, name from students where name like '%Oscar%'"
query2 = "select class, avg(age) from students group by class"

## opening conn
conn = pt.open_connection("config.yml")

## passing the conn object to the query_to_df
df1 = pt.query_to_df(query, conn, 10000)

## passing the conn object to the query_to_df , without to open again
df2 = pt.query_to_df(query2, conn, 10)

## close connection
pt.close_connection(conn)Returns:ID                NAME
0    3298272         Meyer,Oscar
1    2304928         Wilde,Oscar
2    7654321        Grouch,Oscar
.        ...                 ...
128  2234879    De La Hoya,Oscar
129  9872322      Peterson,Oscar
130  9082394       Sanchez,Oscar

[131 rows x 2 columns]Sample config file:database:
    username: "OGROUCH"
    password: "SECR3TPASSWORD"
    sysdba: "n"
    host: >
            (DESCRIPTION =
                    (ADDRESS = (PROTOCOL = TCP)
                               (HOST = servername.myschool.edu)
                               (PORT = 1521)
                    )
                    (CONNECT_DATA = (SERVER = DEDICATED)
                                    (SID = dbname)
                    )
            )If you don’t wish to store your password in the configuration file, you can
omit that line. If a password isn’t present in the configuration file, you
will be prompted for it at runtime.You can choose whether or not to connect with the SYSDBA role. If the yaml
file doesn’t include the “sysdba” property, it will connect like a normal
user.As of version 2.1.0, the _cc methods have been added (query_to_df_cc,
execute_cc, and insert_multiple_cc). These methods get passed a config file
rather than a connection object and take care of opening and closing the
connection for you.
Package: pandaspg
DescriptionThis library contains several functions that allow you to migrate data from a CSV file or Pandas Dataframe into a PostgreSQL database using the libraries Psycopg2 and Pandas. Specifically, it includes functions for:Loading data from a CSV file to a Pandas dataframeMapping Pandas Dataframe columns to their datatypesMapping Pandas Dataframe columns to suitable PostgreSQL datatypesConnect to a PostgreSQL databaseCreating new tables on a PostgreSQL databaseInserting data from a Pandas DataFrame into a table in a PostgreSQL databaseInstallationInstall The Required Dependenciespip install pandas psycopg2Install ThepandaspgPackagepip install pandaspg💡 Make sure you have a PostgreSQL database up and running!UsageI will walk you through a step-by-step example of how to migrate data from aCSVfile into aPostgreSQLdatabase. For demonstration purposes, download theexoplanets_07-04-28.csvfile from my[exoplanets](https://github.com/eadwulf/exoplanets)repo.Download The CSV filewget https://raw.githubusercontent.com/Eadwulf/exoplanets/main/exoplanets_07-04-2023.csv💡 This file contains a total of 34,112 rows and 92 columns with a total size of 28.6 MBImport The LibraryimportpandaspgCreate a Pandasdataframewith theCSVfile datadataframe=pandaspg.csv_to_dataframe('exoplanets_04-07-2023.csv')Generate a dictionary mapping the dataframe columns to their datatypecolumn_datatypes_dict=pandaspg.get_dataframe_column_dtypes_dict(dataframe)Generate a dictionary mapping the dataframe columns to a suitable PostgreSQL datatypepg_column_datatypes_dict=pandaspg.map_pandas_to_postgresql_datatypes(column_datatypes_dict)Connect to an existing and running PostgreSQL databaseconnection=pandaspg.connect_to_postgresql(database='analysis',user='postgres',password='postgres',host='localhost',port=5432)💡 Replace the above parameters as you see fit.Create a PostgreSQL tablepandaspg.create_postgresql_table(connection,'exoplanets_csv',pg_column_datatypes_dict)💡 The parameter `exoplanets_csv` is the name of the database to be created.Insert the from the dataframe to the recently created tablepandaspg.insert_dataframe_into_postgresql(connection,'exoplanets_csv',dataframe)Close the connection with the databaseconnection.close()The Full Exampleimportpandaspgdataframe=pandaspg.csv_to_dataframe('exoplanets_04-07-2023.csv')column_datatypes_dict=pandaspg.get_dataframe_column_dtypes_dict(dataframe)pg_column_datatypes_dict=pandaspg.map_pandas_to_postgresql_datatypes(column_datatypes_dict)connection=pandaspg.connect_to_postgresql(database='analysis',user='postgres',password='postgres',host='localhost',port=5432)pandaspg.create_postgresql_table(connection,'exoplanets_csv',pg_column_datatypes_dict)pandaspg.insert_dataframe_into_postgresql(connection,'exoplanets_csv',dataframe)connection.close()Inspect The ResultsEnter the PostgreSQL promptpsql -U postgres -d analysisList the tables in theanalysisdatabase\dtRetrieve the data from theexoplanets_csvtableSELECT*FROMexoplanets_csv;
Package: pandas-pipe
Purposea collection of pandas DataFrame methods and functions I commonly used decorated withscikit-lego's@log_stepto enable logging within method chainingWanted to use the logging of common pandas methods across projectsInspired from Vincent D. Warmerdam's PyData Eindhoven 2019 talk titledUntitled12.ipynbSee thedocs for pandas pipelines of scikit-legoto view the general use case for@log_stepSeeexamples/demo.ipynbfor a few workflows and links to other relevant SE questionsIn[1]:importpandasaspd;importnumpyasnp;importloggingIn[2]:importpandas_pipeIn[3]:stream_handler=logging.StreamHandler()In[4]:pandas_pipe.logger.addHandler(stream_handler)In[5]:dataf_input=pd.DataFrame(...:{...:"aaa":{0:10,1:11,2:12,3:13},...:"myid":{0:1,1:2,2:3,3:4},...:"num":{0:"1, 2, 3",1:np.nan,2:"1, 2",3:np.nan},...:"text":{0:"aa, bb, cc",1:np.nan,2:"cc, dd",3:"ee"},...:"states":{...:0:"Wyoming; Illinois; New Hampshire",...:1:"Pennsylvania",...:2:"New York",...:3:"Pennsylvania",...:},...:}...:)In[6]:dataf_inputOut[6]:aaamyidnumtextstates01011,2,3aa,bb,ccWyoming;Illinois;NewHampshire1112NaNNaNPennsylvania21231,2cc,ddNewYork3134NaNeePennsylvaniaIn[7]:(...:dataf_input.pipe(pandas_pipe.start_pipeline)...:.pipe(pandas_pipe.explode_setup,columns=["num","text"],delimiter=",")...:.pipe(pandas_pipe.explode_setup,columns=["states"],delimiter=";")...:.pipe(pandas_pipe.explode,column="num",ignore_index=False)...:.pipe(pandas_pipe.explode,column="text",ignore_index=False)...:.pipe(pandas_pipe.explode,column="states",ignore_index=False)...:.pipe(pandas_pipe.drop_duplicates)...:)[start_pipeline(df)]time=0:00:00.000152n_obs=4,n_col=5names=['aaa','myid','num','text','states'][explode_setup(df,columns=['num','text'],delimiter=',')]time=0:00:00.001107n_obs=4,n_col=5delta=(0,0)[explode_setup(df,columns=['states'],delimiter=';')]time=0:00:00.000466n_obs=4,n_col=5delta=(0,0)[explode(df,column='num',ignore_index=False)]time=0:00:00.003697n_obs=7,n_col=5delta=(+3,0)[explode(df,column='text',ignore_index=False)]time=0:00:00.002642n_obs=15,n_col=5delta=(+8,0)[explode(df,column='states',ignore_index=False)]time=0:00:00.002184n_obs=33,n_col=5delta=(+18,0)[drop_duplicates(df)]time=0:00:00.001671n_obs=33,n_col=5delta=(0,0)Out[7]:aaamyidnumtextstates01011aaWyoming01011aaIllinois01011aaNewHampshire01011bbWyoming01011bbIllinois01011bbNewHampshire01011ccWyoming01011ccIllinois01011ccNewHampshire01012aaWyoming01012aaIllinois01012aaNewHampshire01012bbWyoming01012bbIllinois01012bbNewHampshire01012ccWyoming01012ccIllinois01012ccNewHampshire01013aaWyoming01013aaIllinois01013aaNewHampshire01013bbWyoming01013bbIllinois01013bbNewHampshire01013ccWyoming01013ccIllinois01013ccNewHampshire1112NaNNaNPennsylvania21231ccNewYork21231ddNewYork21232ccNewYork21232ddNewYork3134NaNeePennsylvania
Package: pandas-pipeline-graphviz
Pandas pipeline in graphvizPython package to build a nice explanative schema of a data processing pipeline in pandas.It's heavily inspired bydask's.visualizemethod, but improved with 2 useful features:visualize columns names in data nodeshighlight created columns at each taskHere is an example from theexamples folder:InstallationPipInstall with pip:$pipinstallpandas-pipeline-graphvizManual installationInstall manually:git cloneusepython setup.pyUsageDisclaimer⚠️ WARNING — it's a hack!There are no reliable methods in python to get variables names, either as input or as output. The methods used in this package are quitehacky, as discussed in thisstackoverflow thread.To build the graph, this package makes use of:globals()to get the names of input dataframes, doing a comparison between the input dataframes and all the variables available in the global variables.inspect.stack()to get the name of the output dataframe, gathering the code lines calling the function and parsing it to find the output. Currently it supports only single-output transformations.Both methods should be considered as experimental and the behavior of the decorator is expected to break easily if it's not used as presented in theexamples.Conditions for usedo not use several decorators on your function, only this decorator, otherwise it will break the output dataframe name detection throughinspect.stack()use only single output transformation functions, i.e. functions which return only 1 dataframe.ExamplesSeeexamples folderin the repository.
Package: pandasplexos
pandasplexosPandas Plexos wrapperFree software: MIT licenseDocumentation:https://pandasplexos.readthedocs.io.FeaturesTODOCreditsThis package was created withCookiecutterand theaudreyr/cookiecutter-pypackageproject template.History0.1.0 (2016-08-26)First release on PyPI.