Package: delimited2fixedwidth
delimited2fixedwidthConvert files from delimited (e.g. CSV) to fixed width formatHow to run the programHow to install the programFor Linux and Windows, download the latest version fromhere(look under the "Assets" section) and run it on your system, no need to install anything else.The program can also be installed from the Python Package Index:pip install delimited2fixedwidthOr it can be downloaded and ran directly from theDocker Hub:docker run --rm e2jk/delimited2fixedwidth [add parameters here]See belowhow to install from source.Configuration fileIn order for the program to know how to transform your delimited file into a fixed-width file, you will need to provide a configuration file describing the length and type of values expected for your output file.An example configuration file can be found attests/sample_files/configuration1.xlsxA configuration file is a simple Excel.xlsxfile in which each row represents a single field expected in the output file (the fixed-width file), and at least these 3 column headers, i.e. the first line in your Excel file:LengthOutput formatSkip fieldTheLengthvalue is self-explanatory: it represents how long the field will be in the generated fixed-width file. If the value in the input file is shorter than this defined length, it will be padded with0s or spaces, depending on the type of Output format (see next section).TheOutput formatdefines how the input value must be treated and transformed. The following values are supported:IntegerA numeric value that gets padded with0s added to the leftExample: "123" becomes "000123" if a length of 6 is definedDecimalDecimal numbers get sent as "cents" instead of "dollars", rounded to the nearest cent. (yeah, weird explanation -- better have a look at the example...). Also padded with0s added to the left.Example: "123.458" becomes "00012346" if a length of 8 is definedKeep numericstrips all non-numeric characters from an input value and treats the remaining value asIntegerExample: "1-2.3a" becomes "000123" if a length of 6 is definedDateA date to be converted from one format to another. The input value can be sent with either Day or Month as first element or as ISO format YYYYMMDD, and with a slash, dash, dot or no separator. When there is a separator defined, the day and month can omit the leading 0, if need be. See at the top of thetest_main.pyfilefor the full list of supported codes.Examples:"21/06/2020" becomes "20200621" with a format ofDate (DD/MM/YYYY to YYYYMMDD)and a length of 8"6-21-2020" becomes "20200621" with a format ofDate (MM-DD-YYYY to YYYYMMDD)and a length of 8"21062020" becomes "20200621" with a format ofDate (DDMMYYYY to YYYYMMDD)and a length of 8"6.21.2020" becomes "21/06/2020" with a format ofDate (MM.DD.YYYY to DD/MM/YYYY)and a length of 10TimeA time sent as hour:minutes (with or without colon in the input data) will be sent out without the colonExample: "20:06" becomes "2006" if a length of 4 is definedTextThe value gets sent without format changes (such as those outlined above for date and time), with spaces added at the end, on the right of the stringExample: "Hello" becomes "Hello" if a length of 10 is definedFinally, setting the value of theSkip fieldcolumn to "True" allows to send a field as blank in the output file, respecting the field size and padding type:0s or spaces depending on the defined output format.Running the programOpen a Command Line windowcmdand indicate your input file name, output file name and configuration file to use. You can additionally indicate if your input file uses a specific field separator (default is,), textual field wrapper (default is"), or if you want to skip a specific number of header or footer files from your input file.See theProgram help informationsection below for details on how to populate these arguments.An example run of the program could look like this:delimited2fixedwidth.exe --input data\input_file.txt --config data\configuration_file.xlsx --output data\output_file.txt --delimiter "^" --skip-header 1 --skip-footer 1Or it can be ran from theDocker Hub:docker run --rm e2jk/delimited2fixedwidth [add parameters here]If you've installed the program followinghow to install from source, you can run the program withpipenv run python delimited2fixedwidth.py.Program help informationusage: delimited2fixedwidth.py [-h] [--version] [-x] (-i INPUT | -id INPUT_DIRECTORY) [-ie INPUT_ENCODING]
                               (-o OUTPUT | -od OUTPUT_DIRECTORY) [-m] -c CONFIG [-dl DELIMITER] [-q QUOTECHAR] [-sh SKIP_HEADER]
                               [-sf SKIP_FOOTER] [-l LOCALE] [-t TRUNCATE] [-dv DIVERT] [-d] [-v]

Convert files from delimited (e.g. CSV) to fixed width format

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
  -x, --overwrite-file  Allow to overwrite the output file
  -i INPUT, --input INPUT
                        Specify the input file
  -id INPUT_DIRECTORY, --input-directory INPUT_DIRECTORY
                        Specify the input directory from which to process input files
  -ie INPUT_ENCODING, --input-encoding INPUT_ENCODING
                        Specify the encoding of the input files (default: 'utf-8')
  -o OUTPUT, --output OUTPUT
                        Specify the output file
  -od OUTPUT_DIRECTORY, --output-directory OUTPUT_DIRECTORY
                        The directory in which to create the output files
  -m, --move-input-files
                        Move the input files to the output directory after processing. Must be used in conjunction with the `--output-
                        directory` argument.
  -c CONFIG, --config CONFIG
                        Specify the configuration file
  -dl DELIMITER, --delimiter DELIMITER
                        The field delimiter used in the input file (default ,)
  -q QUOTECHAR, --quotechar QUOTECHAR
                        The character used to wrap textual fields in the input file (default ")
  -sh SKIP_HEADER, --skip-header SKIP_HEADER
                        The number of header lines to skip (default 0)
  -sf SKIP_FOOTER, --skip-footer SKIP_FOOTER
                        The number of footer lines to skip (default 0)
  -l LOCALE, --locale LOCALE
                        Change the locale, useful to handle decimal separators
  -t TRUNCATE, --truncate TRUNCATE
                        Comma-delimited list of field numbers for which the output will be truncated at the maximum line length, should the
                        input value be longer than the maximum defined field length. If not set, a field that is too long will cause the
                        script to stop with an error.
  -dv DIVERT, --divert DIVERT
                        Diverts to a separate file the content from rows containing a specific value at a specific place. The format of this
                        parameter is "<field number>,<value to divert on>" (without quotes). This parameter can be repeated several times to
                        support different values or different fields. The diverted content will be saved to a file whose name will be the
                        output filename with "_diverted" added before the file extension.
  -d, --debug           Print lots of debugging statements
  -v, --verbose         Be verboseDevelopment informationHow to install from sourceCreate the environment:cddevel/delimited2fixedwidth/
python3-mpipinstall--userpipenv
pipenvinstallIf you want to develop the script, replace that last command bypipenv install --devActivate the virtual environment:cddevel/delimited2fixedwidth/
pipenvshellYou can also run the script usingpipenv run delimited2fixedwidth.pyinstead ofpython3 delimited2fixedwidth.pywithout having to set up a subshell (which has some problems in Windows, with the history not being accessible with the up arrow)When done:exitUpdate the dependencies:pipenvupdateInstall a new dependencypipenvinstall<package_name>[--dev]Building the executableRun the following command in your virtual environment:pipenv run pyinstaller --onefile delimited2fixedwidth.pyThe executable that gets created in thedistfolder can then be uploaded to Github as a new release.Packaging the source and publishing to the Python Package IndexFollow the instructions mentionedhere, namely:pipenv lock -r > requirements-no-dev.txt
pipenv run python setup.py sdist bdist_wheel
pipx run twine upload dist/*Create the Docker image and publish it to Docker HubRun:docker build -t e2jk/delimited2fixedwidth:latest -t e2jk/delimited2fixedwidth:<version> --rm .to build the Docker image.docker run --rm e2jk/delimited2fixedwidth:<version>to test the Docker image locally.docker push e2jk/delimited2fixedwidth:latestanddocker push e2jk/delimited2fixedwidth:<version>to push the Docker image to Docker Hub.ChangelogThese are the changes brought in each versions:v1.0.11 (2021-07-09)Non-breaking changes:Fix package published on PyPIv1.0.10 (2020-11-21)Non-breaking changes:New--input-encodingargument to specify the encoding of the input files, defaults to 'utf-8'v1.0.9 (2020-11-10)Non-breaking changes:Process all the files in a directory as multiple input files through the new arguments--input-directoryand--output-directoryNew--move-input-filesargument to move the input files to the output directory after processing--outputmoved back as a shared argument (relevant for scripts using delimited2fixedwidth as an imported library), partially reverting a change introduced in v1.0.2Ensure the--overwrite-fileargument is only used with--outputv1.0.8 (2020-11-02)Breaking changes:Date and Time fields now default/pad to 0's instead of spacesNon-breaking changes:The program can now be ran in a Docker container:https://hub.docker.com/r/e2jk/delimited2fixedwidthv1.0.7 (2020-09-30)Non-breaking changes:Support a large number of new date formats. See at the top of thetest_main.pyfilefor the full list of supported codes. Some examples:Date (DD/MM/YYYY to DD/MM/YYYY)Date (YYYYMMDD to DD.MM.YYYY)Date (MM.DD.YYYY to YYYYMMDD)Date (YYYYMMDD to MM-DD-YYYY)New formatKeep numericthat strips all non-numeric characters from an input valueNew--divertargument to divert to a separate file the content from rows containing specific valuesv1.0.6 (2020-09-17)Non-breaking changes:New--truncateargument to specify which fields can be cut when the input value is longer than the defined maximum field lengthNew--localeargument, in case a different Decimal separator is usedFix: Spaces or empty string accepted as valid Integer and Decimal values (interpreted as 0)v1.0.5 (2020-09-16)Non-breaking changes:Support for new date formats:Date (DD-MM-YYYY to YYYYMMDD)Date (MM-DD-YYYY to YYYYMMDD)Date (DD.MM.YYYY to YYYYMMDD)Date (MM.DD.YYYY to YYYYMMDD)Date (DDMMYYYY to YYYYMMDD)Date (MMDDYYYY to YYYYMMDD)v1.0.4 (2020-09-16)Other changes:Reduce the number of dependenciesExclude the development-specific dependencies in the PyPI packagev1.0.3 (2020-09-14)Non-breaking changes:Fix handling of the "Date (MM/DD/YYYY to YYYYMMDD)" date formatOther changes:Changes to the development toolchain and test suitev1.0.2 (2020-09-07)Non-breaking changes:Remove--outputand--overwrite-filefrom the shared arguments (relevant for scripts using delimited2fixedwidth as an imported library)v1.0.1 (2020-09-06)Non-breaking changes:Expose command-line arguments for code that imports this moduleRefactor some of the test suitev1.0.0 (2020-09-05)Non-breaking changes:Refactor code to be able to importprocess()from outside scriptsReturns the number of processed rows and the oldest and most recent dates on a to-be-specified date fieldv0.0.2-alpha (2020-09-04)Breaking changes:FormatDate (DD/MM/YYYY)renamed toDate (DD/MM/YYYY to YYYYMMDD)Non-breaking changes:First version published on PyPISupport for new date format:"Date (MM/DD/YYYY to YYYYMMDD)"New--versionargumentv0.0.1-alpha (2020-08-31)Initial release