Package: combine-csv
combine_csvBased on an idea fromhttps://github.com/ekapope/Combine-CSV-files-in-the-folder/blob/master/Combine_CSVs.py, this small script simply focus on merging CSV/TSV files, by combining either lines or column.ItemProject siteSourcehttps://github.com/gmtsciencedev/combine_csvDocumentationhttps://combine-csv.readthedocs.io/Downloadhttps://pypi.org/project/combine-csv/Keywordspython, csv, merge, combineNB starting from v3 acombine_tsvutility is provided, it behaves exactly the same but default to TAB as a separator.Basic usageThe tool can be used either :in the line mode (default) which use all different CSV to create new lines in a merged CSV,or in the column mode (using flag-c) which use all different CSV to add new columns, using the first column as an index in all files.Line modecombine_csv-i'*.csv'-omy_merged_csv.csvThus if folder contains:1.csvname,age
Jean,23
Paul,122.csvname,age,sex
Jane,19,female
John,74,maleIt will create this file:my_merged_csv.csvname,age,sex
Jean,23,
Paul,12,
Jane,19,female
John,74,maleColumn modecombine_csv-c-i'*.csv'-omy_merged_csv.csvThus if folder contains:1.csvtask_id,name,desc
1,create,create a new object
2,delete,delete an object2.csvtask_id,program
1,create.py
2,delete.py
3,random.pyIt will create this file:my_merged_csv.csvtask_id,name,desc,program
1,create,create a new object,create.py
2,delete,delete an object,delete.py
3,,,random.pyMain optionsSee command linecombine_csv -hfor all options. Here we would like to point the most convenient ones.As you have seen-iis the input selector which takes a python glob.glob pattern (protect it withsinglequotes as in the examples above to prevent shell interpretation), and-ogive the name of the file (which default tocombine.csv)-s --separator:   Change the default field separator from,to whatever you need. For TSV file, say\t(add single quotes around to prevent backslash interpretation by shells, e.g.-s '\t'or-s \\t). This separator will be used to read input files and to write the output file. You can choose to have a different output separator with the-toption which behaves likewise.-a --addname:   Add the name of the input files (without extension). In line mode, this will add a new column namedsource(which name can be changed with the--source-columnoption) containing the name of the files. In column mode, this will add the names to non-index columns preceded by an underscore.SplitBase usageIn version 2.0 a new utility was introduced,split_csv, enabling to do the opposite of merging: splitting. It uses the same flags as above (-s for the input separator, -t for the output separator, -c for column mode).NB starting from v3 asplit_tsvutility is provided, it behaves exactly the same but default to TAB as a separator.For instance, with a sample CSVmy.csvfile like this:x,a,b
1,0,1
2,0,1
3,1,0Using the line mode like this:split_csvmy.csvWill produce three files:my_1.csv:x,a,b
1,0,1my_2.csv:x,a,b
2,0,1my_3.csv:x,a,b
3,1,0While using the column mode with the-coption:split_csv-cmy.csvwill produce two files:my_a.csv:x,a
1,0
2,0
3,1my_b.csv:x,b
1,1
2,1
3,0Controlling new file namingThe file naming uses a default pattern of{input}_{index}.{ext}which uses python F-string syntax. You can provide your own pattern with the-ooption.
Each bracket term is dynamically replaced during splitting:input: input is replaced by the base name of the input file including any path if provided,index: index is replaced by the current name of either the column or the line depending on the mode,ext: ext is replaced by the extension (csvin the above example).In the case of the line-split mode, the pattern may also use one of the column name: the term is replaced by the current value of the column for the line.For instance, in the above example:split_csv-o'my{index}-{a}.csv'my.csvWill create three files:my1-0.csv,my2-0.csvandmy3-1.csvsasNew in v3Asas2csv(resp.sas2tsv) utility is provided to convert SAS file (.sas7bdat) into CSV (resp TSV) files. It relies mostly on pyreadstat library that does all the work here.The utility takes a SAS file and a create a CSV (resp. TSV) file replacing the .sas7bdat extension by .csv (resp .tsv). You may use-ooption to provide your own name.sas2csvmy_sas_file.sas7bdatSAS files contain both column names and column labels. By tradition column names are rather short and upper case in SAS (and may be somewhat cryptic to nowadays standards), as pyreadstat give access to both using-loption will enable to use labels, getting longer, more explicit names in the produced CSV file:sas2csv-lmy_sas_file.sas7bdatpivotNew in v3A pivot transformation is what is called in MS Excel a dynamic table. In the input CSV file, one of the column is holding a variable name, and another one is holding a variable value. In the output TSV file, each variable name becomes a new column of its own, holding the value that was specified in the variable value column:input fileinput.csv:index,variable,value
1,a,11
1,b,12
1,c,13
2,a,21
2,c,23
3,b,32with :pivot_csv-p2-v3input.csva output file calledinput-pivot.csvwill be produced:index,a,b,c
1,11,12,13
2,21,,23
3,,32,-poption defines the position(s) that hold the name of the new columns to create. Here the value is 2, the column namedvariable.-voption defines the position(s) that hold the value(s) of the new cell to create. Here the value is 3, the column namedvalue.The first column is used as an index, the identifier of an object, to which the variable names are attribute. Thus the first line of our input file says that in object1the attributeaholds value11. It means that in our pivoted CSV, we will have in the line starting with1a columnawith value11. In the second line of our input CSV, we are still on the object1thus it will not create a new line in our pivoted CSV, it will add extra columns to the first line. Only the fourth data line introduces a new object, the object2that will create a new line in our pivoted CSV.multiple name columnsYou may define several name columns with-p, using a comma separated list of integer holding the different column positions (starting from 1). This means that variable names are defining combining several columns (all different name parts are aggregated, separated with a space)For instance with input:index,variable1,variable2,value
1,a,1,11
1,b,2,12
1,c,3,13
2,a,1,21
2,c,1,23
3,b,2,32The call usingpivot_csv -p 2,3 -v 4 input.csvwill generate:index,a 1,b 2,c 3,c 1
1,11,12,13,
2,21,,,23
3,,32,,multiple value columnsYou may define several value columns with-v, using a comma separated list of integer holding the different column positions (starting from 1). This means that the the dynamic variable are composite, holding different sub-values. A new column will be created for each of these column per dynamic name, unless they hold no value. Anyway, in that case, the name of the column holding the value will be added to the dynamic name.For instance with input:index,variable,value1,value2
1,a,1,11
1,b,2,12
1,c,3,13
2,a,1,21
2,c,1,23
3,b,2,32The call usingpivot_csv -p 2 -v 3,4 input.csvwill generate:index,a value1,a value 2,b value1,b value2,c value1, c value2
1,1,11,2,12,3,12
2,1,21,,,1,23
3,,,2,32,,using range within -p or -vFor conveniance, instead of enumerating a long list, you may use a range like1-10which is the same as1,2,3,4,5,6,7,8,9,10.
You may combine ranges and simple integer, for instance1,4-7,9-12which is the same as1,4,5,6,7,9,10,11,12You may use partial ranges like4-wich means4,5,6,...ending with the last column of the file.
You may also use negative values as in python,-1meaning the last column,-2the column just before the last column, etc., however you cannot use negative columns in range expressions.extra columns in input fileThe algorithm defines what is to be done with:the first column: this is the index, or the identifier of the object,the name or pivot columns: this define the names of the new columns to create,the value columns: this define the values to add in the new cells.What of other columns? They are interpreted as non dynamic attributes of the object. Non dynamic meaning here that they are not expected to vary if related to the same index or object identifier.For instance:index,name,variable,value
1,dog,weight,1
1,dog,length,2
2,cat,weight,1
2,cat,length,1The call usingpivot_csv -p 3 -v 4 input.csvwill generate:index,name,weight,length
1,dog,1,2
2,cat,1,1If the input file is instead:index,name,variable,value
1,dog,weight,1
1,dog,length,2
2,cat,weight,1
2,bird,length,1The same call would generate a warning:Non pivoted columns hold extra information: column name [#2] contains different information from what was gathered before : 'bird' instead of 'cat', this will be discardedThe output file would not be changed, as thebirdinformation in last line is discarded.To be complete this last input file would also give the same output as before, and without any warning as holding no value at all is not interpreted as a new input:index,name,variable,value
1,dog,weight,1
1,,length,2
2,cat,weight,1
2,,length,1pivot_tsvAs with other utility a sister utility namedpivot_tsvis provided and it behaves exactly the same