Saravanan666/pypi-org huggingface dataset:
Package: csv-remapper
CSV RemapperA GUI/CLI tool for manipulating CSV file dataDescriptionAllows you to remap, extract and combine data from multiple CSV sources.
Key features:Easy to use GUI for creating mapping Templates and processing filesBatch-remap files using command line interfaceCombine data from multiple csv sourcesExtract data using RegexOmit data based on predefined valuesGetting StartedDependenciesPython 3.7 or higherInstallingpip install csv-remapperExecutingDisplay help:csv-remapper -hRun in GUI mode:csv-remapperWorkflowLaunchcsv-remapperin GUI mode and create anOutput Template(Edit → New Template... → Output) by selecting a CSV file.
This file doesn't need to contain any data (apart from the header) and should contain all the columns you wish to see in the final output.Create anInput Templateby selecting a CSV file containing sample data from your source application.Open Template Editor (Edit → Edit Templates), select your Output template and begin remapping data by dragging and dropping table headers.If some values need changing (e.g. date format) do so by selecting the correct In/Out data types (double-click to edit)If you have additional data you wish to insert, which is not present in your main input, you can do so by addingAlias Datatables in the Output section.
Populate the "Alias Values" column - these values will be used for selecting the correct row before data extraction.
As previously, drag and drop table headers in both Output and Input sections to create mappings.All tables are editable via direct data manipulation (double-click) or the context menu (right-click)Once the templates are set up, you can remap your CSV files via GUI or by using the command line, e.g.:csv-remapper -t "Out-Template-Name" -o "//output/dir" -f "file1.csv" "file2.csv" "file3.csv" ...
Package: csvresumable
cssresumableAdding tiny resume function for your csv reading iterator.examples00main.pyimportsysimportosimportrandomfromioimportStringIOfromcsvresumableimportDictReaderdata="""\id,name1,foo2,bar3,boo"""random.seed(int(os.environ.get("SEED","0")))forrowinDictReader(StringIO(data)):n=random.random()ifn<0.5:print("fail{}".format(n),file=sys.stderr)sys.exit(0)print("\t",row["name"])print("finished")resuming until success.SEED=0 python 00main.pyfail 0.420571580830845foobarSEED=1 RESUME=1 python 00main.pyfail 0.13436424411240122SEED=2 RESUME=1 python 00main.pyboofinishedSEED=5 python 00main.pyfoobarboofinishedor setting withresumeoption (run this script, such aspython 01main.py --resume)--- 00main.py	2018-06-16 04:19:51.573578443 +0900+++ 01main.py	2018-06-16 04:20:04.760554103 +0900@@ -4,17 +4,23 @@from io import StringIOfrom csvresumable import DictReader-data = """\+data = """id,name1,foo2,bar3,boo"""+import argparse+parser = argparse.ArgumentParser()+parser.add_argument("--seed", default=0, type=int)+parser.add_argument("--resume", action="store_true")+args = parser.parse_args()+random.seed(int(os.environ.get("SEED", "0")))-for row in DictReader(StringIO(data)):+for row in DictReader(StringIO(data), resume=args.resume):n = random.random()-    if n < 0.5:+    if n > 0.8:print("fail {}".format(n), file=sys.stderr)sys.exit(0)print("\t", row["name"])with multi filesfromcsvresumableimportDictReaderfiles=["a.csv","b.csv"]forfinfiles:r=DictReader(f)forrowinr:do_something(row)moreexamples/01multisource
Package: csv-scavenger
CSV-ScavengerCSV-Scavenger is a simple CSV parser for Python which can detect the delimiter automatically and can parse CSV files with quoted fields. It can also detect the header row and the start and end of the data rows automatically. For most use cases, you can simply callsv.read_csv(filename)and get a pandas DataFrame back.See thecsv-scavenger/example_csvsdirectory for some example CSV files which are parsed correctly by CSV-Scavenger.Installationpip install csv-scavengerUsageimportscavengerassv# Read a CSV file into a pandas DataFramedf=sv.read_csv('data.csv')
Package: csv-schema
CSV Schema Definition and ValidationInstallpipinstallcsv-schemaUsageusage: csv-schema [-h] [--version] {validate-config,validate-csv,generate-config} ...

CSV Schema

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit

Commands:
  {validate-config,validate-csv,generate-config}
    validate-config     Validates the CSV schema JSON configuration file.
    validate-csv        Validates a CSV file against a schema.
    generate-config     Generate a CSV schema JSON configuration file.DefinitionThe CSV file schema definition is defined in a JSON file in the following format.File Definition:{"name":null,"description":null,"filename":{"regex":null},"columns":[]}PropertyDescriptionnameThe name of the schema.descriptionThe description of the schema.filenameProperties for the name of the CSV filename to validate.columnsList of column definitions.filename{"regex":null}PropertyDescriptionregexRegular expression to validate the name of the CSV file being validated.Column Definitions:string{"type":"string","name":null,"required":true,"null_or_empty":false,"regex":null,"min":null,"max":null}PropertyDescriptiontypeThe column type.nameThe name of the column.requiredWhether or not the column is required in the file.null_or_emptyWhether or not the value can be null (missing) or an empty string.regexRegular expression to validate the column value.minThe minimum length of the string. null for no limit.maxThe maximum length of the string. null for no limit.integer{"type":"integer","name":null,"required":true,"null_or_empty":false,"regex":null,"min":null,"max":null}PropertyDescriptiontypeThe column type.nameThe name of the column.requiredWhether or not the column is required in the file.null_or_emptyWhether or not the value can be null (missing) or an empty string.regexRegular expression to validate the column value.minThe minimum value. null for no limit.maxThe maximum value. null for no limit.decimal{"type":"decimal","name":null,"required":true,"null_or_empty":false,"regex":null,"min":null,"max":null,"precision":2}PropertyDescriptiontypeThe column type.nameThe name of the column.requiredWhether or not the column is required in the file.null_or_emptyWhether or not the value can be null (missing) or an empty string.regexRegular expression to validate the column value.minThe minimum value. null for no limit.maxThe maximum value. null for no limit.precisionThe decimal point precision.enum{"type":"enum","name":null,"required":true,"null_or_empty":false,"values":[]}PropertyDescriptiontypeThe column type.nameThe name of the column.requiredWhether or not the column is required in the file.null_or_emptyWhether or not the value can be null (missing) or an empty string.valuesFixed set of constants.DevelopmentSetuppipenv--three
pipenvshell
makepip_install
makebuild
makeinstall_localSeeMakefilefor all commands.Update README DefinitionsThe schema definition is auto generated by runningscripts/generate_md_help.py.
Copy/paste the output into the README.md file.TestingCreate and activate a virtual environment:Run the tests:make test
Package: csv-schema-inference
Csv Schema InferenceA tool to automatically infer columns data types in .csv filesCheck the article here:Building a Schema Inference Data Pipeline for Large CSV filesInstalling csv-schema-inference🔧pipinstallcsv-schema-inferenceLooking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting csv-schema-inference
  Downloading csv_schema_inference-0.0.9-py3-none-any.whl (7.3 kB)
Installing collected packages: csv-schema-inference
Successfully installed csv-schema-inference-0.0.9Importing csv-schema-inference library⚡fromcsv_schema_inferenceimportcsv_schema_inferenceSetting csv-schema-inference configuration✍#if the inferred data type is INTEGER and there is a presence of FLOAT on the results , then the result will be FLOATconditions={"INTEGER":"FLOAT"}csv_infer=csv_schema_inference.CsvSchemaInference(portion=0.9,max_length=100,batch_size=200000,acc=0.8,seed=2,header=True,sep=",",conditions=conditions)pathfile="/content/file__500k.csv"Run inference🏃aprox_schema=csv_infer.run_inference(pathfile)Showing the approximate data type inference for each column🔍csv_infer.pretty(aprox_schema)0
	name
		id
	type
		INTEGER
	nullable
		False
1
	name
		full_name
	type
		STRING
	nullable
		True
2
	name
		age
	type
		INTEGER
	nullable
		False
3
	name
		city
	type
		STRING
	nullable
		True
4
	name
		weight
	type
		FLOAT
	nullable
		False
5
	name
		height
	type
		FLOAT
	nullable
		False
6
	name
		isActive
	type
		BOOLEAN
	nullable
		False
7
	name
		col_int1
	type
		INTEGER
	nullable
		False
8
	name
		col_int2
	type
		INTEGER
	nullable
		False
9
	name
		col_int3
	type
		INTEGER
	nullable
		False
10
	name
		col_float1
	type
		FLOAT
	nullable
		False
11
	name
		col_float2
	type
		FLOAT
	nullable
		False
12
	name
		col_float3
	type
		FLOAT
	nullable
		False
13
	name
		col_float4
	type
		FLOAT
	nullable
		False
14
	name
		col_float5
	type
		FLOAT
	nullable
		False
15
	name
		col_float6
	type
		FLOAT
	nullable
		False
16
	name
		col_float7
	type
		FLOAT
	nullable
		False
17
	name
		col_float8
	type
		FLOAT
	nullable
		False
18
	name
		col_float9
	type
		FLOAT
	nullable
		False
19
	name
		col_float10
	type
		FLOAT
	nullable
		False
20
	name
		test_column
	type
		FLOAT
	nullable
		FalseChecking schema values for specific columns✔result=csv_infer.get_schema_columns(columns={"test_column"})csv_infer.pretty(result)20
	_name
		test_column
	types_found
		INTEGER
			cnt
				406130
		FLOAT
			cnt
				50964
	nullable
		False
	type
		FLOATExplore all possible data types for a specific columns✅result=csv_infer.explore_schema_column(column="test_column")csv_infer.pretty(result)20
	name
		test_column
	types_found
		INTEGER
			88.85043339006856
		FLOAT
			11.149566609931437
	nullable
		FalseContributing and FeedbackAny ideas or feedback about this repository?. Help me to improve it.AuthorsCreated byRamses Alexander Coraspe ValdezCreated on 2022LicenseThis project is licensed under the terms of the MIT License.
Package: csvschemavalidation
csvschemavalidationcsvschemavalidation is an implementation ofCSV Schemain Python.This project is at Alpha version and still under heavy development.Example>>>fromcsvschemavalidation.checkerimportValidator>>># demo.csv:...# id,name,value...# 1,Ann,"5"...# 2,Ben,"10"...# 3,Tom,"14">>>schema={...'fields':[...{...'name':'value',...'type':'number',...'multipleOf':5...}...]...}>>>v=Validator(csvfile='demo.csv',schema=schema)>>>v.validate()Traceback(mostrecentcalllast):...<ValidationError:'Value 14.0 is not multiple of 5';columnname:value;rownumber:3>Note that the validator does not check if the CSV format fits the dialect defined in schema correctly. For example, if wrong delimiter present in schema, validator might read the whole line as one cell.InstallationpipinstallcsvschemavalidationRequirementsPython 3.5 or aboveTODODocumentation and ExamplesOptional headerDecide whether to rely on Python's built-in int and float function to convert valuesTests
Package: csvs-convert
CSVS ConvertConverts CSV files into XLSX/SQLITE/POSTGRESQL/PARQUET fast.Installpipinstallcsvs_convertDocsFull DocumentaionAimsThorough type guessing of CSV columns, so there is no need to configure types of each field. Scans whole file first to make sure all types in a column are consistent. Can detect over 30 date/time formats as well as JSON data.Quick conversions/type guessing (uses rust underneath). Uses fast methods specific for each output format:copyfor postgresPrepared statements for sqlite using c API.Arrow reader for parquetWrite only mode for libxlsxwriterTries to limit errors when inserting data into database by resorting to "text" if type guessing can't determine a more specific type.When inserting into existing databases automatically migrate schema of target to allow for new data (evolveoption).Memory efficient. All csvs and outputs are streamed so all conversions should take up very little memory.Gather stats and information about CSV files into datapacakge.json file which can use it for customizing conversion.DrawbacksCSV files currently need header rows.Whole file needs to be on disk as whole CSV is analyzed therefore files are read twice.Conversion DocsThis is the python library, providing bindings to therust library.Contribute on githubUsage From CSV files.import csvs_convert

#sqlite
csvs_convert.csvs_to_sqlite("sqlite.db", ["file.csv"])
#postgres
csvs_convert.csvs_to_postgres("postgresql://user:postgres@localhost/db", ["file.csv"])
#parquet
csvs_convert.csvs_to_parquet("output", ["file.csv"])
#xlsx
csvs_convert.csvs_to_xlsx("output.xlsx", ["sqlite.db"])Usage from datapackageA datapackage is a file that contains metadata about the tables its specification is describedhere.To generatedatapackage.jsonfile you can use:csvs_convert.csvs_to_datapackage('path/to/datapackage.json', ["fixtures/large/csv/data.csv"])Other tools can also generate these files.You can use this file and alter it as needed. Mostly it is useful if you want to use the same schema across multiple files, as it will save time not having to do the type guessing for every file.When referring to a datapackage you can either reference:Adatapackage.jsonfile.A datapackage directory containing adatapackage.jsonfile. e.g./a/datapackage/dirA zip file containing adatapackage.jsonfile. e.g.my_datapackage.zipExamples:import csvs_convert

#sqlite
csvs_convert.datapackage_to_sqlite("sqlite.db", "path/to/datapackage.json")
#postgres
csvs_convert.datapackage_to_postgres("postgresql://user:postgres@localhost/db", "path/to/datapackage.json")
#parquet
csvs_convert.datapackage_to_parquet("path/to/directory", ["sqlite.db"])
#xlsx
csvs_convert.datapackage_to_xlsx("output.xlsx", "path/to/datapackage.json")
Package: csvsdataset
csvsdatasetcsvsdatasetis a Python library designed to simplify the process of working with multiple CSV files as a single dataset. The primary functionality is provided by theCsvsDatasetclass in thecsvsdataset.pymodule.This was written by ChatGPT4 as mentionedhere. Issues will be cut and paste into a session. It is an experiment in semi-autonomous code maintenance.InstallationTo install thecsvsdatasetlibrary, simply run:pipinstallcsvsdatasetUsagefrom csvsdataset.csvsdataset import CsvsDataset
    
    # Initialize the CsvsDataset instance
    dataset = CsvsDataset(folder_path="path/to/your/csv/folder",
                          file_pattern="*.csv",
                          x_columns=["column1", "column2"],
                          y_column="target_column")
    
    # Iterate over the dataset
    for x_data, y_data in dataset:
        # Your processing code here
        pass
    
    # Access a specific item in the dataset
    x_data, y_data = dataset[42]Memory frugalityOnly data from a small number of csv files are maintained in memory. The
rest is discarded on a LRU basis. This class is intended for use
when a very large number of data files exist which cannot be loaded into
memory conveniently.
Package: csvsed
A stream-oriented CSV modification tool. Like a stripped-down “sed”
command, but for tabular data.TL;DRInstall:$pipinstallcsvsedUse:# given a sample CSV$catsample.csvEmployeeID,Age,Wage,Status8783,47,"104,343,873.83","All good, but nowhere to go."2003,32,"98,878,784.00",A-OK# modify that data with a series of `csvsed` pipes$catsample.csv\|csvsed-cWages/,//g\# remove commas from the Wage column|csvsed-cStatus'y/A-Z/a-z/'\# convert Status to all lowercase|csvsed-cStatus's/.*(ok|good).*/\1/'\# restrict to keywords 'ok' & 'good'|csvsed-cAge'e/xargs -I {} echo "{}*2" | bc/'# double the Age columnEmployeeID,Age,Wage,Status8783,94,104343873.83,good2003,64,98878784.00,okInstallation$pipinstallcsvsedUsage and ExamplesInstallation of thecsvsedpython package also installs thecsvsedcommand-line tool. Usecsvsed--helpfor all command
line options, but here are some examples to get you going. Given the
input filesample.csv:Employee ID,Age,Wage,Status
8783,47,"104,343,873.83","All good, but nowhere to go."
2003,32,"98,878,784.00",A-OKRemoving thousands-separators from the “Wage” column using the “s”
(substitute) modifier:$catsample.csv|csvsed-cWages/,//gEmployeeID,Age,Wage,Status8783,47,104343873.83,"All good, but nowhere to go."2003,32,98878784.00,A-OKConvert/extract some text using the “s” (substitute) and “y”
(transliterate) modifiers:$catsample.csv|csvsed-cStatus's/^All (.*),.*/\1/'\|csvsed-cStatus's/^A-(.*)/\1/'\|csvsed-cStatus'y/a-z/A-Z/'EmployeeID,Age,Wage,Status8783,47,"104,343,873.83",GOOD2003,32,"98,878,784.00",OKSquare the “Age” column using the “e” (execute) modifier:$catsample.csv|csvsed-cAge'e/xargs -I {} echo "{}^2" | bc/'EmployeeID,Age,Wage,Status8783,2209,"104,343,873.83","All good, but nowhere to go."2003,1024,"98,878,784.00",A-OKThat, however, called the external program for each column (quite
inefficient with large data sets)… so let’s do that more
efficiently, with a “continuous” mode program. Given the followingid2name.pyprogram which takes a CSV on STDIN with a single column
(an employee ID) and writes a CSV to STDOUT with the IDs converted to
names:#!/usr/bin/env pythonimportsys,csvkittable={'8783':'ElfenKyng','2003':'Stradivarius'}# NOTE: *not* using csvkit's reader because it reads-ahead# causing problems since this must be stream-oriented...writer=csvkit.CSVKitWriter(sys.stdout)whileTrue:item=sys.stdin.readline()ifnotitem:breakitem=item.strip()writer.writerow([table[item]ifitemintableelseitem])sys.stdout.flush()Then the following will efficiently convert the ‘Employee ID’ column
to names:$catsample.csv|csvsed-c'Employee ID''e|./id2name.py|c'EmployeeID,Age,Wage,StatusElfenKyng,47,"104,343,873.83","All good, but nowhere to go."Stradivarius,32,"98,878,784.00",A-OK
Package: csvsheet
csvsheetSimple formula support for CSVDescriptionThe aim of this project is to provide simple math capabilities to CSV files.
The capabilities are reduced to be able to usepythonformulas in a cell.
The formulas are evaluated usingeval, so a basic sanitazion is done to avoid major risks, to do so, onlymathlibrary can be used as well assint,floatand+|-|*||(|).Command line usageThe command line requires a single compulsory arugment, the CSV fileto process:csvsheet <file.csv> -o <output.csv> -d <delimiter> -q <quotechar> -m <equation_delimiter> -v -vv
csvsheet --helpThe input file is required, if ‘-’ is used, stdin is used.
The output file is optional, if not provided, the output will be printed to stdout.
The delimiter is also optional, if not provided, the default delimiter is,.
The quotechar is also optional, if not provided, the default quotechar is“.
The equation delimiter is also optional, if not provided, the default equation delimiter is=.The-vand-vvflags are optional, they are used to increase the verbosity of the output.Read theextended docsfor extra information.ExampleThe following example shows how to use the command line:csvsheet example.csvIf the contents ofexample.csvare:item,result
Great,=1+2
Not so great,=math.pow(2, 3)The output will be:item,result
Great,3
Not so great,8.0Python usageThe python usage is very simple, it is only required to import thecsvsheetmodule and call therunfunction:import csvsheet

csvsheet.run(["tests/test_a.csv"])Making Changes & ContributingThis project usespre-commit, please make sure to install it before making any
changes:pip install pre-commit
cd csvsheet
pre-commit installIt is a good idea to update the hooks to the latest version:pre-commit autoupdateDon’t forget to tell your contributors to also install and use pre-commit.NoteThis project has been set up using PyScaffold 4.4.1. For details and usage
information on PyScaffold seehttps://pyscaffold.org/.
Package: csvshuf
Shuffle cells by column in CSV files.UsageShuffle the first column of each row of foobar.csvcsvshuf -c1 foobar.csvShuffle the third column of each row of foobar.csv using Sattolo’s
algorithmcsvshuf -c3 -s foobar.csvShuffle all columns of foobar.csvcsvshuf foobar.csvShuffle all the columns but the first of foobar.csvcsvshuf -C1 foobar.csvShuffle the first and third columns of the first ten lines of foobar.csvhead -10 foobar.csv | csvshuf -c 1,3Shuffle the first and third columns of the pipe-delimited foobar.csvcsvshuf -c1,3 -d “|” foobar.csvShuffle the first and third columns of the tab-delimited foobar.csvcsvshuf -c 1,3 -t foobar.csvShuffle the first three columns of the pipe-delimited foobar.csv; output
will be comma-delimitedcsvshuf -c 1,2,3 -d “|” -o , foobar.csvShuffle the first three columns of the comma-delimited foobar.csv;
output will be pipe-delimitedcsvshuf -c 1,2,3 -o “|” foobar.csvShuffle the first two columns of the comma-delimited, pipe-quoted
foobar.csvcsvshuf -c 1,2 -d “,” -q “|” foobar.csvInstallationpip install csvshufAuthorPere Orgapere@orga.cat, 2016.Originally forked from csvcut (https://gist.github.com/bycoffe/187278).
Package: csv-shuffler
csv-shufflerA tool to automatically Shuffle lines in a .csv filesInstallingcsv_shuffler🔧pipinstallcsv-shufflerLooking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting csv-shuffler
  Downloading csv_shuffler-0.0.3-py3-none-any.whl (2.9 kB)
Installing collected packages: csv-shuffler
Successfully installed csv-shuffler-0.0.3Importingcsv_shufflerlibrary ⚡fromcsv_shufflerimportcsv_shufflerSettingcsv_shufflerconfiguration ✍shuffler=csv_shuffler.ShuffleCSV(input_file='/content/sample_data/mnist_train_small.csv',header=True,batch_size=20000)Run shuffler 🏃shuffler.shuffle_csv()19999Contributing and FeedbackAny ideas or feedback about this repository?. Help me to improve it.AuthorsCreated byRamses Alexander Coraspe ValdezCreated on 2022LicenseThis project is licensed under the terms of the MIT License.
Package: csv-similarity
CSV-SimilarityIntroA toolkit to get or remove similar items from the csv fileExamplefromcsv_similarity.similarityimport*get_similar(input_path=f'data/list_company_news1.csv',similarity=0.8,save_path=f'data/similarity_report1.csv',# stopwords_path=f'{root_path}/stopwords/stopwords',stopwords_path='',analyze_field='title')remove_similar(similarity_report_path=f'data/similarity_report1.csv',input_csv_path=f'data/list_company_news1.csv',output_path=f'data/list_company_news_without_similar.csv',)LicenseThecsv-similaritytoolkit is developed byDonghua Chen.
Package: csvsimple
“csvsimple” provides a simple tool to handle CSV data.See example on Github :https://github.com/denis-beurive/csvsimple/blob/master/examples/ex.pyRequirementThis module requires Python 3 or greater.Documentationhttp://packages.python.org/csvsimple/index.htmlSee this module on pypi.python.orghttp://pypi.python.org/pypi?:action=display&name=csvsimpleInstallation from sourceGet the source on Github:https://github.com/denis-beurive/csvsimplepython setup.py sdistpython setup.py install
Package: csvsort
CSV SortFor sorting CSV files on disk that do not fit into memory. The merge sort algorithm is used to break up the original file into smaller chunks, sort these in memory, and then merge these sorted files.Example usage>>>fromcsvsortimportcsvsort>>># sort this CSV on the 5th and 3rd columns (columns are 0 indexed)>>>csvsort('test1.csv',[4,2])>>># sort this CSV with no header on 4th column and save results to separate file>>>csvsort('test2.csv',[3],output_filename='test3.csv',has_header=False)>>># sort this TSV on the first column and use a maximum of 10MB per split>>>csvsort('test3.tsv',[0],max_size=10,delimiter='\t')>>># sort this CSV on the first column and force quotes around every field (default is csv.QUOTE_MINIMAL)>>>importcsv>>>csvsort('test4.csv',[0],quoting=csv.QUOTE_ALL)InstallSupports python 2 & 3:$pipinstallcsvsort$pip3installcsvsort