Saravanan666/pypi-org huggingface dataset:
Package: dataframe-pointer
DataFrame pointerIn many cases you may need an element of a pandas DataFrame to represent complex data.
Pandas supports using DataFrames as elements of DataFrames, but doing so breaks a lot of its functionality.
One way to deal with this is to place a string or other identifier as the dataframe element and have a function or dict that maps this identifier onto the data it represents.
However this makes creating new values a pain - you need to name it, create a dataframe for it, and add this dataframe to the lookup function/dict.
If making up new values for e.g. testing purposes, this adds additional steps and imports to each module.An alternative is to create an object that contains the dataframe but is recognizable as a unique, hashable object by pandas.
dataframe_pointer does this for you using DataFrame accessors.Simplyimport dataframe_pointerand every dataframe will now have a.pointer()method which returns an object that has a .df attribute which is a copy of the original dataframe.
Any two identical dataframes will have the same pointer, as in the following test case to understand what I mean:def test_df_pointer_differs_by_df():
    df1 = pd.DataFrame({'a': range(1000)})
    df2 = pd.DataFrame({'a': range(200)})
    df3 = pd.DataFrame({'1234': range(4000)})
    point1 = df1.pointer()
    point2 = df2.pointer()
    point3 = df3.pointer()
    pd.testing.assert_frame_equal(point1.df, df1)
    assert point1 != point2
    assert point2 != point3
    assert point1 != point3
    assert point1 == df1.pointer()
    assert point1 == df1.copy().pointer()df.pointer(name)creates a named dataframe pointer. If a pointer for an identical dataframe already exists, that pointer will be returned and not renamed.For large dataframes this may be prohibitively inefficient, I'm planning some features to deal with those cases.Planned future featuresdf.pointer(name, hash=False)will use the pointer's name to identify the pointer instead of the dataframe's contents. This will be vastly more efficient for large dataframes, but requires the user to be careful not to create pointers with the same name for multiple dataframes.
Therefore, it will raise an error if the same name is passed twice. Find a pointr by name usingdf.pointer.find(name)ordataframe_pointer.find(name).dataframe_pointer.create_pointer(df_initializer, name)will likewise hash a pointer by name, but can be passed a callable instead of a dataframe.
This callable will return the dataframe in question and will only be evaluated when needed, saving memory and processing time in some cases.
Package: dataframe-quickview
DataFrame QuickViewDataFrame QuickViewis a Python package that extends pandas DataFrame functionality to easily display and visualize DataFrames in a web-based environment. This package is an experiment in paired programming with GPT-4. It is built using Flask and allows users to view paginated DataFrames and interactively generate histograms based on the selected columns.FeaturesPhase 1Extend pandas DataFrame withquickview()methodDisplay paginated DataFrame in a web browserCreate an interactive dropdown and button combination populated with DataFrame columnsGenerate histograms based on the selected column when the button is pressedPhase 1 Technical OverviewFlask server: Represents the backend server running the Flask application, which serves the paginated DataFrame and processes the histogram data.DataFrame display: The component in the web browser that shows the paginated DataFrame.Column selection: The dropdown and button combination that allows users to select a column for generating the histogram.Histogram generation: The component responsible for creating a histogram based on the selected column.Interactions:Flask server sends the paginated DataFrame to the DataFrame display.User selects a column in the Column selection component.User clicks the button in the Column selection component.Flask server receives the selected column and processes the histogram data.Histogram generation component receives the data and displays the histogram.UsageInstall the package using pip:pip install dataframe-quickviewImport the package and use thequickview()method on a pandas DataFrame object:importpandasaspdfromdataframe_quickviewimportquickviewdata={'A':[1,2,3,4,5],'B':[2,4,6,8,10],'C':[3,6,9,12,15]}df=pd.DataFrame(data)df.quickview()This will start the Flask server and open the browser to view the paginated DataFrame and interactive histogram.ContributingWe welcome contributions to this project. However, please note that all code added to the project should be written primarily by Language Models (LLMs) to maintain the experimental nature of this project.LicenseThis project is licensed under the MIT License.
Package: dataframer
dataframerTries to load any file into a pandas DataFrame,
with a minimum of configuration,
and a focus on bioinformaticsExamplesTypically, you’ll read a file from disk (open('my-file.txt', 'rb')),
but a byte stream is simpler here.>>> from io import BytesIO
>>> from dataframer import dataframer
>>> from pandas import set_option

>>> set_option('display.max_columns', None)

>>> bytes = b'a,b,c,z\n1,2,3,foo\n4,5,6,bar'
>>> stream = BytesIO(bytes)Default behavior is to strip non-numeric values after the first column.>>> df_info = dataframer.parse(stream)
>>> df_info.data_frame
   b  c
a      
1  2  3
4  5  6
>>> df_info.label_map is None
TrueAlternatively, they can be preserved in place...>>> df_info = dataframer.parse(stream, keep_strings=True)
>>> df_info.data_frame
   b  c    z
a           
1  2  3  foo
4  5  6  bar
>>> df_info.label_map is None
True... or they can be used to compose more meaningful row labels.>>> df_info = dataframer.parse(stream, relabel=True)
>>> df_info.data_frame
   b  c
a      
1  2  3
4  5  6
>>> df_info.label_map
{1: 'foo / 1', 4: 'bar / 4'}Alternatively, the first column can also be treated as data.>>> df_info = dataframer.parse(stream, col_zero_index=False)
>>> df_info.data_frame
   a  b  c
0  1  2  3
1  4  5  6
>>> df_info.label_map is None
TrueIf you don't need the whole file, but instead only want the first
row for column information:>>> df_info = dataframer.parse(stream, first_row_only=True)
>>> df_info.data_frame
   b  c
a      
1  2  3
>>> df_info.label_map is None
TrueSingle column lists are given an implicit header:>>> bytes = b'banana\napple\npear'
>>> stream = BytesIO(bytes)
>>> df_info = dataframer.parse(stream)
>>> df_info.data_frame
     item
0  banana
1   apple
2    pearRelease processIn your branch updateVERSION.txt, using semantic versioning: When the PR
is merged, the successful Travis build will push a new version to pypi.
Package: dataframe-service
Pandas DataFrame Service (DFS)Features* Simple, ~100 lines multi-threaded file cache implementation
* Key-value store for Panda DataFrames with basic index querying
* Fixed budget memory consumption w/ LRU eviction
* Supports updates on files and dataframes
* Simple TCP client/server interface w/ client-side connection poolingLimitations1. Currently does not support replication, though the file system can be (e.g. NAS)UsageInstall$pip3installdataframe-serviceStart server$dfs_server
Servingon0.0.0.0port8000withmaxmemory1073741824atrootdirectory<currentdir>Run DFS REPL$rlwrapdfs_repl
WelcometoDFSREPL
author:BrianGuarraci
repo:https://github.com/briangu/dfs
crtl-ctoquit

?>stats{"memory":{"used":"0","free":"1073741824","max":"1073741824"}}
Package: dataframe-sql
dataframe_sqlis aPythonpackage that translates SQL syntax into operations onpandasDataFrames, a functionality which is not available in the central pandas
package.Installationpipinstalldataframe_sqlUsageIn this simple example, a DataFrame is read in from a csv and then using the query
function you can produce a new DataFrame from the sql query.frompandasimportread_csvfromdataframe_sqlimportregister_temp_table,querymy_table=read_csv("some_file.csv")register_temp_table(my_table,"my_table")query("""select * from my_table""")The package currently only supportspandasbut there are plans to supportdask,rapids, andmodinin the future.SQL SyntaxThe SQL syntax fordataframe_sqlis exactly the same as the syntax insql_to_ibis, its underlying package.You can find the full SQL syntaxhereWhy use dataframe_sql?While there are other packages that accomplish the goal of using SQL with pandas
DataFrames, all other packages such aspandasqlactually use a database on the
backend which defeats the purpose of usingpandasto begin with. In the case ofpandasqlwhich uses SQLite, this can result in major performance bottlenecks.dataframe_sqlactually performs native pandas operations in memory on DataFrames,
which avoids conflicts that may arise from using external databases.
Package: dataframetodb
DataframeToDBThis project is a personal non-profit project rather than accepting donations to buy more coffee and carry out more open source projects!If you want to donate:https://ko-fi.com/juanretamalesIntroductionDataframeToDBis an improved way to upload pandas dataframes to Microsoft SQL Server, MySQL, PostgreDB and support for other type of database.DataframeToDBtakes advantage of SQLAlchemy. This allows for a much lighter weight import for writing pandas dataframes to db server.When to use:When you need save dataframes, for example when scraping many tablesYou need shared a database estructure for use in proyectsWhen you need save multi excels (ported to dataframed) in a databaseWhen you consider use FastAPI (With the SQLAlchemy compatibility) (Work in progress)When you need create a estructure of table for any reasonInstallationFor instalation you can copy the dataframetodb folder into your proyect from github, orpipinstalldataframetodbRequirementsWritten for Python 3.8+numpy>=1.20.2pandas>=1.2.4python-dateutil>=2.8.1SQLAlchemy>=1.4.9driver for db conectionExample - Save dataframe into databaseimportpandasaspdimportdataframetodbfromdataframetodbimportTable,refactorfromdatetimeimportdatetimeimportosnametable="nameTable"engine=dataframetodb.create_engine('sqlite:///{}.sqlite'.format(nametable))#create engine for use SQLAlchemydf=pd.read_csv("./dataset/data.csv")# Get the DataFramedf=refactor(df)# Returns a dataframe with the correct dtype compatible with DataframeToDB.table_class=Table(name=nametable,df=df)#create Table instancetable_class.toDb(df,engine,'append')#insert data in database, in this example sqliteHow to workThe Table of DataframeToDB is a custom class who generate a SQLAlchemy Table when you call getTable function.USAGEFor more information, you can viewthe documentationingithubor view de documentation of code.FAQWhat databases does DataframeToDB support?The same as SQLAlchemy, for now they are SQLite, Postgresql, MySQL, Oracle, MS-SQL, Firebird, Sybase and others. But you should checkthis link.why use json and not pickle?It really cost me to decide, while pickle is faster, json is more transparent and it is easier to make modifications and transfers of files between projects, also it is expected that json can be used not only in tables, if not, in the entire database.why did you decide to create this library?For scrapping projects it is very tedious to be configuring tables, I wanted something more automatic, I found one but it was only compatible with MS-SQL, and in projects that could not afford that database I decided to create this and create things that I felt were missing.When launch the 1.X release?When y tested all function i like for a table, my plans is 0.X i create all function for transport a dataframe to db and work, in the 1.X i tried to create database class who extends the functions for work with Database.Cats or Dogs?Pandas!!! ...ba dum tss...
Package: dataframe-to-image
dataframe_to_imagePython package to convert a Pandas DataFrame to a Image that could be used to share dataframe (as an image) in sheets, notes and slides.Example usage:importpandasaspdfromdataframe_to_imageimportdataframe_to_image#creating the dataframedf=pd.DataFrame()df['date']=['2021-06-01','2021-06-02','2021-06-03']df['calories']=[1950,2100,1500]df['sleep hours']=[9,7.5,8.2]df['gym']=[True,False,False]#converting to imagedataframe_to_image.convert(df,visualisation_library='matplotlib')Once you have the image ready, you can right click on the image and click 'Save image as' to save the image of your dataframe (when using matplotlib). For Plotly, you can just click on the camera icon on the top right to save your image.
Package: dataframetranslationscoring
Installation:pipinstalldataframetranslationscoringUsage:Given two folder paths , the translation_quantitative_scoring() function will return a dictionary with translation quality scores.
Example:importpandasaspdfromdataframetranslationscoring.scoringimporttranslation_quantitative_scoringmetrics=translation_quantitative_scoring(original_dataframe_path="folder1/*.csv",translated_dataframe_path="folder2/*.csv")This will print the following:Percentageofwordstranslatedvsavailable:X%Percentageofcharacterstranslatedvsavailable:X%Percentageofnon-nulltranslations:X%Totalnumberofcharacterstobetranslated:XTotalnumberofwordstranslated/available:X/XTotalnumberofcharacterstranslated/available:X/Xand return a dictionary with the following keys:{'words_percentage':'X%','characters_percentage':'X%','non_null_percentage':'X%','total_characters':X,'total_words_translated/available':'X/X','total_characters_translated/available':'X/X'}
Package: dataframeXconverter
Dataframe to CSV,HTML and XML converterDescriptionThis is a package that converts your dataframe to csv,xml and html.This package has three funtions toConvert dataframe to csvConvert dataframe to xmlConvert dataframe to htmlUsageRun Following command to install.pip install dataframeXconverterExample of How to use.Make sure the dataframe you are passing is not an empty one.import dataframeXconverter

# to convert dataframe to csv
df_to_csv(dataframe, "filename.csv")

# to convert dataframe to xml
df_to_csv(dataframe, "filename.xml")

# to convert dataframe to html
df_to_csv(dataframe, "filename.html")Error and Exception HandlingI have included some error and exception handling in the code,if you want then you can log them. Im just throwing(raise) it ,you have to catch it in your script.# in csv function

    if not isinstance(dataframe, pd.DataFrame):
       raise AttributeError('dataframe arg in df_to_csv() is not a dataframe')
   if not isinstance(filename, str) or not filename.endswith('.csv'):
       raise NameError('Please Provide Valid CSV File Name')

# in xml funtion

   if not isinstance(dataframe, pd.DataFrame):
       raise AttributeError('dataframe arg in df_to_xml() is not a dataframe')
   if not isinstance(filename, str) or not filename.endswith('.xml'):
       raise NameError('Please Provide Valid XML File Name')

# in html function

   if not isinstance(dataframe, pd.DataFrame):
       raise AttributeError('dataframe arg in df_to_html() is not a dataframe')
   if not isinstance(filename, str) or not filename.endswith('.html'):
       raise NameError('Please Provide Valid HTML File Name')
Package: dataframing
MotivationThedataframingpackage provides useful functions to use dataframes.Data transformationThe main goal is to allow you to transforme a dataframe structure into
another in a way which is easy to use, understand how structures are
connected and allows you to work with typing.It assumes that you have aProtocoldefining your dataframes (which
by the way is a convenient thing to do!). For example:>>>fromtypingimportProtocol>>>importdataframingasdfr>>>>>>classOriginal(Protocol):...last_name:str...first_name:str>>>>>>classModified(Protocol):...full_name:strNow we build a transformer that connectsOriginalandModified>>>withdfr.morph(Original,Modified)as(ori2mod,source,target):...target.full_name=dfr.wrap("{},{}".format,source.last_name,source.first_name)And now is ready to use!>>>row=dict(last_name="Cleese",first_name="John")>>>ori2mod.transform_record(row){'full_name':'Cleese, John'}Notice that we are demonstrating this with a dictionary but it will work this
with a dataframe row, or a full dataframe (or iterable of dicts).>>>data=[...dict(last_name="Cleese",first_name="John"),...dict(last_name="Gilliam",first_name="Terry")...]>>>ori2mod.transform_collection(data)[{'full_name':'Cleese, John'},{'full_name':'Gilliam, Terry'}]If you are going to use a particular function a lot,
you can wrap it once and use it multiple times. This also helps to keep
the converter visually clean.>>>fullnamer=dfr.wrap("{},{}".format)>>>withdfr.morph(Original,Modified)as(ori2mod,source,target):...target.full_name=fullnamer(source.last_name,source.first_name)To show case how to create two columns from one, we are going to build the reverse
transformer.>>>defsplitter(s:str)->tuple[str,str]:...part1,part2=s.split(",")...returnpart1.strip(),part2.strip()>>>namesplitter=dfr.wrap(splitter)>>>withdfr.morph(Modified,Original)as(mod2ori,source,target):...target.last_name,target.first_name=namesplitter(source.full_name)>>>>>>row=dict(full_name="Cleese, John")>>>mod2ori.transform_record(row){'last_name':'Cleese','first_name':'John'}Input/OutputYou can also use it to save and load data.>>>dfr.save(my_dataframe,"example.xlsx")# doctest: +SKIP>>>df=dfr.load("example.xlsx")# doctest: +SKIPWhy using this instead of the standardpandas.to_excel?savedoes two extra things:Stores the metadata stored inmy_dataframe.attrsfrom/into another sheet.Calculates a hash for the data and metadata and store it in the
metadata sheet.Loads will compare the data content with the stored hash. This behaviour is
useful for data validation, but can be disable withuse_hashkeyword argument.Another useful pair of functions areload_many,save_many>>>dfr.save_many(dict(raw_data=raw_data,processed_data=processed_data),"example.xlsx")# doctest: +SKIP>>>dfdict=dfr.load_many("example.xlsx")# doctest: +SKIPin which the input and output are dictionaries that allows you to group into
a single excel file multiple dataframes.InstallationJust install it using:pipinstalldataframing
Package: datafs
DataFS is a package manager for data. It manages file versions, dependencies, and metadata for individual use or large organizations.Configure and connect to a metadataManagerand multiple dataServicesusing a specification file and you’ll be sharing, tracking, and using your data in seconds.Free software: MIT licenseDocumentation:https://datafs.readthedocs.io.FeaturesExplicit version and metadata management for teamsUnified read/write interface across file systemsEasily create out-of-the-box configuration files for usersTrack data dependencies and usage logsUse datafs from python or from the command linePermissions handled by managers & services, giving you control over user accessUsageFirst,configure an API. Don’t worry. It’s not too bad. Check out thequickstartto follow along.We’ll assume we already have an API object created and attached to a service called “local”. Once you have this, you can start using DataFS to create and use archives.$datafscreatemy_new_data_archive--description"a test archive"createdversionedarchive<DataArchivelocal://my_new_data_archive>$echo"initial file contents">my_file.txt$datafsupdatemy_new_data_archivemy_file.txt$datafscatmy_new_data_archiveinitialfilecontentsVersions are tracked explicitly. Bump versions on write, and read old versions
if desired.$echo"updated contents">my_file.txt$datafsupdatemy_new_data_archivemy_file.txt--bumpversionminoruploadeddatato<DataArchivelocal://my_new_data_archive>.versionbumped0.0.1-->0.1.$datafscatmy_new_data_archiveupdatedcontents$datafscatmy_new_data_archive--version0.0.1initialfilecontentsPin versions using a requirements file to set the default version$echo"my_new_data_archive==0.0.1">requirements_data.txt$datafscatmy_new_data_archiveinitialfilecontentsAll of these features are available from (and faster in) python:>>>importdatafs>>>api=datafs.get_api()>>>archive=api.get_archive('my_new_data_archive')>>>witharchive.open('r',version='latest')asf:...print(f.read())...updatedcontentsIf you have permission to delete archives, it’s easy to do. Seeadministrative toolsfor tips on setting permissions.$datafsdeletemy_new_data_archivedeletedarchive<DataArchivelocal://my_new_data_archive>Seeexamplesfor more extensive use cases.Installationpip install datafsAdditionally, you’ll need a manager and services:Managers:MongoDB:pip install pymongoDynamoDB:pip install boto3Services:Ready out-of-the-box:localsharedmountedzipftphttp/httpsin-memoryRequiring additional packages:AWS/S3:pip install botoSFTP:pip install paramikoXMLRPC:pip install xmlrpclibRequirementsFor now, DataFS requires python 2.7. We’re working on 3x support.TodoSeeissuesto see and add to our todos.CreditsThis package was created byJustin SimcockandMichael Delgadoof theClimate Impact Lab. Check us out ongithub.Major kudos to the folks atPyFilesystem. Thanks also toaudreyrfor the wonderfulcookiecutterpackage, and toPyup, a constant source of inspiration and our silent third contributor.
Package: datafuncs
Data FuncsFunctions for Data ProcessingFunctions.mean(data) - Returns the mean of a list.grt(data, value) - Returns the mean of a list for values greater than specified.grtEqual(data, value) - Returns the mean of a list for values greater than or equal to specified.less(data, value) - Returns the mean of a list for values less than specified.lessEqual(data, value) - Returns the mean of a list for values less than or equal to specified.notEqual(data, value) - Returns the mean of a list for values not equal to specified.outlier(data, low, high) - Removes the outliers from a list.imputate(data, low, high) - Replaces outliers in a list with the mean of the filtered list..median(data) - Returns the median of a list.mode(data) - Returns the mode of a list.freq(data, value) - Returns the Amount of times that a value occurs in the list data.Range(data) - Returns the range of the list.replace(data, old, new) - Replaces every old value in the list with the new value.gen(length, low, high, type) - Generates a list of random values of the type specified that is length long - Ignore the low and high values if you are not generating a list of integers.all(data) - Returns everything the script knows about a list.Help() - Prints a list of the above functions and their explanationsCreated by Liam Leonard & Fraser Woodward