Saravanan666/pypi-org huggingface dataset:
Package: datatools
Data tools (or, for friends,d) is a command-line-first data analysis
library. The goal of the library is to make data-wrangling tasks easy
and promote code reuse. Each script is supposed to do one thing well,
following the principles of the UNIX philosophy of “doing one thing and
doing it well”.Design principlesThe command line is your friend.An extra command (and pipe) is better than another argument.CSV and JSON get priority over other formats.Always think of scalability and memory issues. Favour streaming algorithms.DRY.List of toolsAll the tools start with the abbreviated name ofdatatools, which isd.dbyrowperforms operations between elements of the same row.dcomputeperforms operations which involve one field of multiple rows (for example, the average).dformat(TO DO) translate existing formats to a standard CSV format.djoinperforms equality joins between tables, both in semi-streaming and full-streaming fashions.drandomgenerates random integer and float values.duniquekeeps unique values.djsonexplorerloads a JSON document into a Python interpreter.
Package: data-toolset
data-tools(et)data-toolset is designed to simplify your data processing tasks by providing a more user-friendly alternative to the traditional JAR utilities like avro-tools and parquet-tools. With this Python package, you can effortlessly handle various data file formats, including Avro and Parquet, using a simple and intuitive command-line interface.InstallationPython 3.8, Python 3.9 and 3.10 are supported and tested (to some extent).python-mpipinstall--userdata-toolsetUsage$data-toolset-h
usage:data-toolset[-h]{head,tail,meta,schema,stats,query,validate,merge,count,to_json,to_csv,to_avro,to_parquet,random_sample}...

positionalarguments:{head,tail,meta,schema,stats,query,validate,merge,count,to_json,to_csv,to_avro,to_parquet,random_sample}commandsheadPrintthefirstNrecordsfromafiletailPrintthelastNrecordsfromafilemetaPrintafile'smetadataschemaPrinttheAvroschemaforafilestatsPrintstatisticsaboutafilequeryQueryafilevalidateValidateafilemergeMergemultiplefilesintoonecountCountthenumberofrecordsinafileto_jsonConvertafiletoJSONformatto_csvConvertafiletoCSVformatto_avroConvertafiletoAvroformatto_parquetConvertafiletoParquetformatrandom_sampleRandomlysamplerecordsfromafileExamplesPrint the first 10 records of a Parquet file:$data-toolsetheadmy_data.parquet-n10shape:(1,7)┌───────────┬─────┬──────────┬────────┬──────────────────────────┬────────────────────────────┬──────────────────┐
│character┆age┆is_human┆height┆quote┆friends┆appearance│
│---┆---┆---┆---┆---┆---┆---│
│str┆i64┆bool┆f64┆str┆list[str]┆struct[2]│
╞═══════════╪═════╪══════════╪════════╪══════════════════════════╪════════════════════════════╪══════════════════╡
│Alice┆10┆true┆150.5┆Curiouserandcuriouser!┆["Rabbit","Cheshire Cat"]┆{"blue","small"}│
└───────────┴─────┴──────────┴────────┴──────────────────────────┴────────────────────────────┴──────────────────┘Query a Parquet file using a SQL-like expression:$data-toolsetquerymy_data.parquet"SELECT * FROM 'my_data.parquet' WHERE height > 165"shape:(2,7)┌─────────────────┬─────┬──────────┬────────┬───────────────────────┬────────────────────────────────────┬───────────────────┐
│character┆age┆is_human┆height┆quote┆friends┆appearance│
│---┆---┆---┆---┆---┆---┆---│
│str┆i64┆bool┆f64┆str┆list[str]┆struct[2]│
╞═════════════════╪═════╪══════════╪════════╪═══════════════════════╪════════════════════════════════════╪═══════════════════╡
│MadHatter┆35┆true┆175.2┆I'mlate!┆["Alice"]┆{"green","tall"}│
│QueenofHearts┆50┆false┆165.8┆Offwiththeirheads!┆["White Rabbit","King of Hearts"]┆{"red","average"}│
└─────────────────┴─────┴──────────┴────────┴───────────────────────┴────────────────────────────────────┴───────────────────┘Merge multiple Avro files into one:data-toolsetmergefile1.avrofile2.avrofile3.avromerged_file.avroConvert Avro file into Parquet:data-toolsetto_parquetmy_data.avrooutput.parquetConvert Parquet file into JSON:data-toolsetto_jsonmy_data.parquetoutput.jsonContributingContributions are welcome! If you have any suggestions, bug reports, or feature requests, please open an issue on GitHub.TODOoptimizations [TBD]benchmarking
Package: data-tools-traders-club
Data ToolsData Science oriented tools packageFeaturesConnection with differents databases to perform queries
-- MySQL
-- MongoDBData CleaningInstalationYou can install this package usingpip install data-tools-traders-clubUsage# Connection
## MySQL
### Import
from data_tools.connection.mysql.api import MySQL

### Credentials
username = os.getenv('MYSQL_USERNAME')
password = os.getenv('MYSQL_PASSWORD')
host = os.getenv('MYSQL_HOST')

### Class Instantiation
mysql_in =  MySQL(username, password, host)

### Query
schema = 'SCHEMA'
table = 'TABLE'

query = f"""SELECT * FROM {schema}.{table} LIMIT 100"""

df = mysql_in.select_query(query, schema)

## MongoDB
### Import
from data_tools.connection.mongodb.api import MongoDB

### Credentials
username = os.getenv('MONGODB_USERNAME')
password = os.getenv('MONGODB_PASSWORD')
host_string = os.getenv('MONGODB_HOST_STRING')

### Class Instantiation
mongodb_in = MongoDB(username, password, host_string)

### Pipeline
database = os.getenv('MONGODB_DATABASE_TEST')
collection = os.getenv('MONGODB_COLLECTION_TEST')
pipeline = [
    {
        '$project': {
            'id': '$id'
        }
    }, {
        '$limit': 10
    }
]

df = mongodb_in.select_pipeline(database, collection, pipeline)

# tools
## DataCleaning
### Import
from data_tools.tools.api import DataCleaning

### Class Instantiation
clean = DataCleaning()

### Rename DataFrame (snake_case)
df = clean.rename_dataframe_snakecase(df)

### Rename DataFrame (CamelCase)
df = clean.rename_dataframe_camelcase(df)Project Strutureroot
├───data_tools
│   ├───connection
│   │   ├───mongodb
│   │   │   └── api.py
│   │   └───mysql
│   │         └── api.py
│   └───tools
│               └── api.py
├───tests 
│   └───connection
│         ├───mongodb
│         │   └── test_api.py
│         └───mysql
│               └── test_api.py
├───venv ## Virtual enviroment (Local)
├───.env ## environmental variables (Local)
├───.gitignore
├───Authors.rst
├───CONTRIBUITING.md
├───History.md
├───LICENSE
├───Makefile
├───MANIFEST.in
├───README.md
├───requirements.txt
└───setup.py
Package: data-toolz
data-toolzThis repository contains reusable python code for data projects.The motivation for this project was to create a package which allows to abstract dataset read/write operations fromdestination type (local,s3,<tbd...>) andtarget file type (delimiter-separated values,jsonlines,parquet)This would allow to write code easily transferable between local and cloud applications.installationpipinstalldata-toolzusagedatatoolz.filesystem.FileSystemclass gives you an abstraction for accesing both local and remote object using the well know pythonicopen()interface.fromdatatoolz.filesystemimportFileSystemforfs_typein("local","s3"):fs=FileSystem(name=fs_type)# common pythonic interface for both local and remote file systemswithfs.open("my-folder-or-bucket/my-file",mode="wt")asfo:fo.write("Hello World!")datatoolz.io.DataIOclass gives you a versatile Reader/Writer interface for handling of typical data files (jsonlines,dsv,parquet)importpandasaspdfromdatatoolz.ioimportDataIOdf=pd.DataFrame({"col1":[1,2,3],"col2":["a","b","c"]})dio=DataIO()# defaults to "local" FileSystem# write as parquetdio.write(dataframe=df,path="my-file.parquet",filetype="parquet")dio.read(path="my-file.parquet",filetype="parquet")# write as gzip-compressed jsonlinesdio.write(dataframe=df,path="my-file.json.gz",filetype="jsonlines",gzip=True)dio.read(path="my-file.json.gz",filetype="jsonlines",gzip=True)# write as delimiter-separated-values in multiple partitionsdio.write(dataframe=df,path="my-file.tsv",filetype="dsv",sep="\t",partition_by=["col1"])dio.read(path="my-file.tsv",filetype="dsv",sep="\t")# write output in multiple chunks per partitiondio.write(dataframe=df,path="my-prefix",filetype="dsv",sep="\t",partition_by=["col1"],suffix=["chunk01.tsv","chunk02.tsv"])dio.read(path="my-prefix",filetype="dsv",sep="\t")datatoolz.logging.JsonLoggeris a wrapper logger for outputting JSON-structured logsfromdatatoolz.loggingimportJsonLoggerlogger=JsonLogger(name="my-custom-logger",env="dev")logger.info(msg="what is my purpose?",meaning_of_life=42){"logger": {"application": "my-custom-logger", "environment": "dev"}, "level": "info", "timestamp": "2020-11-03 18:31:07.757534", "message": "what is my purpose?", "extra": {"meaning_of_life": 42}}It can also be used to decorate functions and log their execution detailsfromdatatoolz.loggingimportJsonLoggerlogger=JsonLogger(name="my-custom-logger",env="dev")@logger.decorate(msg="my-custom-log",duration=True,memory=True,my_value="my-value",output_length=lambdax:len(x))defmy_func(x,y):returnx+y,x*yprint(my_func(42,2)){"logger": {"application": "my-custom-logger", "environment": "dev"}, "level": "info", "timestamp": "2021-03-24 18:10:47.054703", "message": "my-custom-log", "extra": {"function": "my_func", "memory": {"current": 432, "peak": 432}, "duration": 2.5980000000203063e-06, "my_value": "my-value", "output_length": 2}}
(44, 84)
Package: datatorch
DataTorch PythonDataTorch CLI and Python API libary for programmatic access.InstallationNote: Newer versions (>=0.4.8.2) of DataTorch client uselibmagicfor more
accurate mimetype guessing. This falls back to guessing via extension. Consider
installinglibmagicif you have issues with file types uploaded to DataTorch.pipinstalldatatorchDevelopmentpython3-mpipinstall--editable.
pip3install-rrequirements.txtVSCode ConfigurationFormatterOpen your VSCode settings, by goingCode -> Preferences -> Settings. Search
for "python formatting provider" and select "black" from the dropdown menu.
Package: datatosk
DatatoskPython library for fetching data from different sourcesUsageDatatosk reads configuration from theenvironment variables.MySQLProvide particular enviroment variables:MYSQL_HOST_[SOURCE_NAME]=
MYSQL_PORT_[SOURCE_NAME]=
MYSQL_USER_[SOURCE_NAME]=
MYSQL_PASS_[SOURCE_NAME]=
MYSQL__DB_[SOURCE_NAME]=GoogleBigQueryProvide particular enviroment variables:GBQ_PROJECT_ID_[SOURCE_NAME]=
Package: datatour
DataTourSeeing is important.datatour- allows you to see your data in it's native dimension.
Currently implemented as aplotlyscatter plot projected from it's original dimension in the 2D on the screen with timeline animation inspired by GrandTour and common sense.InstallationAvailable via pip:pip install datatourUsageIf you have array of feature vectorsf:shape(shape)==(n_smpl, n_dim), you can create data tour object, and display it:from datatour import DataTour as dt

ndv = dt(f)
ndv.display()By default selects randomlyn_subsample=500samples for efficiency reason.To visualize vector fieldvfof the same dimension (in the same feature space):ndv = dt(f, vf)
ndv.display_quiver(color='z_scaled')Also check examples:dt().display()ndv = dt(example='sphere', n_subsample=0)
ndv.display(color='z_scaled')LicenceDistributed under GNU GPLv3 licence
Package: datatracker
DatatrackerDatatracker is a basic logging Python package that keeps track of files and code within a Project. Each script is logged as an entry and input and output datafiles are recorded in order. Datatracker is able to manage versioning of both files and scripts, and is able to identify the most up-to-date version.At the moment, this Python package is still in alpha, and I may include changes to both UI and file format that may be breaking.InstallationTo install, run the following command:pipinstallgit+ssh://git@github.com/TarjinderSingh/datatrackerUsageNew entriesFor an entry,tagis a unique identifier to  the script in question and should be clear what the general purpose and output of the script is. (ie Merge is not what we want to see here)descriptionneeds to be one or two sentences equivalent of the Git commit message that thoroughly describes the general purpose and output of the script.categoryindicates the general step of analysis the script belongs to.moduleis the sub-category for which the script belongs to. Typecategory_templatein interactive Python for an idea of the appropriate categories and modules are.For a InputFile or OutputFile,tagis a unique identifier to the File in question and should be clear what the general purpose and output of the script is. (ie Merge is not what we want to see here).descriptionfor a file is a one or two sentences equivalent of the Git commit message that thoroughly describe the general purposes of the File at hand.fromdatatrackerimport*tr=Tracker()os.environ['VERSION']='0.1.0'entry=Entry(tag='filter-common-variants',description='Filtering common variants in new GWAS data set.',category='Processing',module='Variant QC')infile=entry.add(InputFile(tag='raw-plink-file',path='gs://bucket/raw-plink-file.bed',description='Raw PLINK file.'))outfile=entry.add(OutputFile(tag='filt-plink-file',path='gs://bucket/raw-plink-file.bed',description='Filtered PLINK file.'))tr.save(entry)View existing entriesfromdatatrackerimport*tr=Tracker()tr.tableUse existing entries for pipelineinfile=entry.add(InputFile(entry_tag='filter-common-variants',tag='raw-plink-file',database=tr))Filter and remove# filter to entrytr.filter(tr.entry.tag_version=='import-array_0.1.6')# remove entrytr.remove(tr.entry.tag_version=='import-array_0.1.6')Pandas and Exceldf=tr.explode()df=tr.explode('filt-plink-file')df=tr.to_pandas()df=tr.tabledf.to_excel('spreadsheet.xlsx')Data artifactsinfile=entry.add(InputFile(path='gs://checkpoint-cache/tmp/1.bed'))LicenseMIT License (see repository)MaintainerTJ Singh @ tsingh@broadinstitute.org
Package: datatrail-cli
Datatrail CLIInstall·Docs🛠️ InstallationInstall from PYPIpipinstalldatatrail-cliClone the repositorygitclonehttps://github.com/CHIMEFRB/datatrail-clicddatatrail-cli
pipinstall.⚙️  ConfigurationLocal# Create Datatrail config filedatatrailconfiginit--sitelocal# Ensure valid CADC Certificate existscadc-get-cert-u[username]CANFARThe Configuration steps at CANFAR are similar to the local configuration.
One notable exception is that after installation, thedatatrailcommand
may not be in the PATH. It is installed to~/.local/bin, in which case
you may need to add it to your PATH or prepend it to the follwing commands.# Create Datatrail config filedatatrailconfiginit--sitecanfar# Ensure valid CADC Certificate existscadc-get-cert-u[username]CHIME# Create Datatrail config filedatatrailconfiginit--sitechime# Ensure valid CADC Certificate existscadc-get-cert-u[username]
Package: datatrails-receipt-scitt
DataTrails Scitt Receipt Verification in pythonPython support for the offline verification of scitt receipts for DataTrails events.Notice: This work is based on thedraftstandarddraft-birkholz-scitt-receipts-02. This work and the upstream standard will evolve. We make no current promise of api stabilityPython SupportThis package currently is tested against Python versions 3.8,3.9, 3.10 and 3.11.The current default version is 3.8 - this means that this package will not
use any features specific to versions 3.9 and later.After End of Life of a particular Python version, support is offered on a best effort
basis. We may ask you to update your Python version to help solve the problem,
if it cannot be reasonably resolved in your current version.
Package: data-transformation-library
Data Transformation LibraryThe Data Transformation Library is a Python library that provides a set of functions for common data transformation tasks.
It is designed to simplify the process of handling and processing data, making it easier for developers to work with various types of data structures.InstallationYou can install the library using pip:pip install data-transformation-libraryFeaturestranspose2dThe transpose2d function switches the axes of a 2D tensor, making it easy to transform rows into columns and vice versa.import data_transformation_libraryinput_matrix = [[1, 2, 3], [4, 5, 6]]transposed_matrix = data_transformation_library.transpose2d(input_matrix)window1dThe window1d function creates overlapping or non-overlapping 1D windows from an input array. This is useful for tasks like time series analysis and data segmentation.import data_transformation_libraryimport numpy as npinput_array = np.array([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])windows = data_transformation_library.window1d(input_array, size=3, shift=2, stride=1)concolution2dThe convolution2d function performs 2D convolution between an input matrix and a kernel. This is a fundamental operation in image processing and feature extraction.import data_transformation_libraryimport numpy as npinput_matrix = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])kernel = np.array([[0, 1], [2, 3]])stride = 1result = data_transformation_library.convolution2d(input_matrix, kernel, stride=stride)Link to the pip packagehttps://pypi.org/project/data-transformation-library/
Package: datatransformationslibrary
Data Transformations LibraryThis library provides essential functions for manipulating and transforming data structures commonly used in data science and machine learning.Table of ContentsInstallationFunctionstranspose2dwindow1dconvolution2dRunning TestsContributionLicenseInstallationClone the repositorygitclonegit@github.com:TuringCollegeSubmissions/smizak-DE2.1.gitNavigate to the directorycdsmizak-DE2.1Install necessary packagespipinstall-rrequirements.txtInstall from pypiAlternatively one can install the package from version published on pypi.
package can be found onhttps://pypi.org/project/datatransformationslibrary/pipinstalldatatransformationslibraryFunctionstranspose2dTransposes a 2D matrix.Usage:fromsrc.DataTransformationsLibraryimporttranspose2dresult=transpose2d(input_matrix)Arguments:input_matrix (list[list[float]]): A 2D matrix to transpose.Returns:A list representing the transposed matrix.Raises:ValueError: If the input matrix is empty.ValueError: If all rows of the matrix do not have the same length.window1dGenerates windows of the specified size, shift, and stride from the input array.Usage:fromsrc.DataTransformationsLibraryimportwindow1dresult=window1d(input_array,size,shift=1,stride=1)Arguments:input_array (Union[list, np.ndarray]): 1D list or numpy array to generate windows from.size (int): The size of each window.shift (int, optional): Number of positions to move the window each iteration. Defaults to 1.stride (int, optional): The step between consecutive elements within each window. Defaults to 1.Returns:A list of windows.Raises:ValueError: If input_array is not a list or 1D numpy array.ValueError: If input_array is not 1D.convolution2dPerforms a 2D convolution operation on the input matrix with a specified kernel.Usage:fromsrc.DataTransformationsLibraryimportconvolution2dresult=convolution2d(input_matrix,kernel,stride=1)Arguments:input_matrix (np.ndarray): A 2D input matrix for convolution.kernel (np.ndarray): A 2D kernel for convolution.stride (int, optional): The step size to use when applying the kernel. Defaults to 1.Returns:A matrix resulting from the convolution operation.Raises:ValueError: If stride is less than or equal to 0.Running TestsTo ensure the integrity of this library, a comprehensive set of tests are provided. To run the tests:Ensure pytest is installedpipinstallpytestNavigate to the tests directory# Navigate to tests directorycdtest# Run the testspytest
Package: data-transformation-tc
data_transformation_tcOverviewdata_transformation_tc is a Python library designed to streamline common data transformations required in machine learning and data science workflows, such as transpose a matrix, create time series windows, or apply 2D convolution.FeaturesTranspose: Quickly transpose any 2D matrix with the transpose2d function.Time Series Windowing: Generate windows for your 1D data arrays or lists with the window1d function. Customize window size, shift, and stride as per your needs.2D Convolution: Apply 2D convolution on matrices using the convolution2d function. This function computes the cross-correlation of an input matrix with a specified kernel.InstallationTo install the data_transforms package, use pip:pip install data-transformation-tcUsageA quick example to get you started:fromdata_transformation_tcimporttranspose2d,window1d,convolution2dimportnumpyasnpfromtypingimportUnion,List# Transpose a matrixmatrix=[[1.0,2.0,3.0],[4.0,5.0,6.0],[7.0,8.0,9.0]]transposed_matrix=transpose2d(matrix)print(transposed_matrix)# Create time series windowsinput_data=np.array([0,1,2,3,4,5,6])windows=window1d(input_data,size=3,shift=1,stride=2)print(windows)# Apply 2D convolutioninput_matrix=np.array([[1,2,3],[4,5,6],[7,8,9]])kernel=np.array([[1,0],[0,-1]])conv_result=convolution2d(input_matrix,kernel)print(conv_result)
Package: data-trans-functions
Data Trans FunctionsDescriptionThedata-trans-functionspackage provides utility functions to perform operations on 2D matrices
and 1D arrays. With features like 2D matrix transposition, 1D windowing, and 2D convolution, it
simplifies common operations while ensuring type safety.Featurestranspose2d: Transposes a 2D matrix, effectively switching its axes.window1d: Generates a series of 1D sliding windows from an input array.convolution2d: Computes the 2D convolution of an input matrix using a given kernel.InstallationTo install the package viapip, run:
pip install data-trans-functionsUsageHere are brief examples for each function:fromdata_trans_functionsimporttranspose2d,window1d,convolution2dimportnumpyasnp# Using transpose2d:matrix=[[1,2,3],[4,5,6]]print(transpose2d(matrix))# Returns: [[1, 4], [2, 5], [3, 6]]# Using window1d:arr=[1,2,3,4]print(window1d(arr,2,shift=2))# Returns: [[1, 2], [3, 4]]# Using convolution2d:input_matrix=np.array([[1,2,3],[4,5,6],[7,8,9]])kernel=np.array([[1,0],[0,-1]])print(convolution2d(input_matrix,kernel))# Returns: [[3, 2], [3, 2]]RecommendationsIt's recommended to use these functions in combination with a type checker like mypy for safer usage.
Package: data-trans-lib
Data transformation libraryThis python library consists of 3 common data transformation functions:transpose2dwindow1dconvolution2dInstallationYou can install this package using pip:pip install data-trans-libThis library is supported on Python 3.9 and above.How to usetranspose2DTransposes a 2-D list - switches rows and columns.Parameters:input_matrix- is a list of lists of real numbers to transpose.Example usagefromdata_trans_lib.transformationsimporttranspose2dinput_matrix=[[1.0,2.0,3.0],[4.0,5.0,6.0]]transposed_matrix=transpose2d(input_matrix)# Output:# [[1.0, 4.0],#  [2.0, 5.0],#  [3.0, 6.0]]window1DExtracts specified size windows from a 1D list or numpy array. The shift for the starting position of the window and the stride between consecutive windows could be specified.Parameters:input_array- is a list or 1D Numpy array of real numbers from which windows are extracted.size- is a positive integer that determines the size (length) of the window.shiftis a positive integer that determines the shift (step size) between different windows.strideis a positive integer that determines the stride (step size) within each window. Default shift and stride values equals to .Example usagefromdata_trans_lib.transformationsimportwindow1dinput_array=[1,2,3,4,5,6,7,8,9,10]size=3shift=2stride=2windows=window1d(input_array,size,shift,stride)# Output:# [[1, 3, 5],#  [3, 5, 7],#  [5, 7, 9]]convolution2dPerforms 2D convolution between the input matrix and the kernel with the specified stride, and it returns the resulting 2D NumPy array. The convolution operation is implemented by sliding the kernel over the input matrix with the specified stride and computing the element-wise multiplication and summation.Parameters:input_matrix- is a 2D Numpy array of real numbers to convolve.kernel- is a 2D Numpy array of real numbers.strideis a positive integer that determines the stride (step size) within each window. Default stride value equals to 1.Example usageimportnumpyasnpfromdata_trans_lib.transformationsimportconvolution2dinput_matrix=np.array([[0,1,2],[3,4,5],[6,7,8]])kernel=np.array([[0,1],[1,0]])convolution=convolution2d(input_matrix,kernel)# Output:# [[ 4,  6],#  [10, 12]