Package: eparse
eparseDescriptionExcel spreadsheet crawler and table parser for data extraction
and querying.FeaturesCommand-line interfaceRecursive Excel file discoverySub-tabular data extraction (logical tables)SQLite and PostgreSQL database interfacesCLI query toolSummary data metricsInstallationTo install eparse, you can use pip and the latest version on PyPI:$pipinstalleparseOr you can clone this repo and install from source:$gitclonehttps://github.com/ChrisPappalardo/eparse.git$cdeparse$pipinstall.If you plan to use the postgres interface, you also need to install
the postgres packagepsycopg2. Instructions can be foundhere.
This package is optional, and you can use the other interfaces
such as theSQLite3interface without having to installpsycopg2.The easiest way to install thepsycopg2package for your
particular environment may be to install the pre-compiled
binary driver as follows:$pipinstallpsycopg2-binaryIf you see an error while trying to use a postgres endpoint such
aspostgres://user:pass@host:port/my_dbthat mentions the
postgres driver is missing, then you know you haven’t properly
installed (and compiled)psycopg2.Usageeparse can be used as either a python library or from the command-line.
You can view supported CLI commands and usage with--helpas follows:$eparse--helpUsage:eparse[OPTIONS]COMMAND[ARGS]...excelparserOptions:-i,--inputTEXTinputsource-o,--outputTEXToutputdestination-f,--fileTEXTfile(s)ordir(s)totarget-d,--debugusedebugmode-l,--loosefindtablesloosely-r,--recursivefindfilesrecursively-t,--truncatetruncatedataframeoutput-v,--verboseincreaseoutputverbosity--helpShowthismessageandexit.Commands:migratemigrateeparsetableparseparsetable(s)foundinsheetfortarget(s)queryqueryeparseoutputscanscanforexcelfilesintargetYou can also use eparse from python like so:fromeparse.coreimportget_df_from_fileprint([tablefortableinget_df_from_file('myfile.xlsx')])102DatePrincipalRepaymentDatePrincipalRepayment10344834700757.67900444926430013.1483031044492671957.77610845016100576.12780810545016147578.1926245107898008.3400951064510732801.36307245199841656.13896...ScanTo scan one or more directories for Excel files with descriptive
information, you can use thescancommand like so:$eparse-v-f<path_to_files>scanIncrease the verbosity with additional flags, such as-vvv, for
more descriptive information about the file(s), including sheet names.ParseExcel files can be parsed as follows:$eparse-v-f<path_to_files>parseThis mode will list each table found in each Excel file to the command-line.
This mode is useful for initial discovery for parseable data.eparse uses a simple algorithm for identifying tables.  Table “corners”
are identified as cells that contain empty cells above and to the right
(or sheet boundaries).  A densely or sparsely populated 2x2 table must
follow in order for data to be extracted in relation to that cell.
eparse will automatically adjust for rowspan labels and empty table
corners and the dense vs. sparse criterion can be controlled with
the--looseflag.eparse was written to accomodate various types of output formats and
endpoints, includingnull:///,stdout:///,sqlite3:///db_name,
andpostgres://user:password@host:port/db_name.nullThis mode is useful for validating files and generating descriptive
info, and is the default.  The command above with-vis an example
of this mode, which lists out the tables found.stdoutThis mode is good for viewing data extracted from Excel files in the
console.  For example, you could view all tables found inSheet1with the following command:$eparse-f<path_to_files>-ostdout:///parse-s"Sheet1"eparse usespandasto handle table data.  You can view larger tables without truncation
using the-tflag as follows:$eparse-t-f<path_to_files>-ostdout:///parse-s"Sheet1"Data in table format is useful for human viewing, but a serialized
form is better for data interfacing.  Serialize your output with
the-zflag as follows:$eparse-t-f<path_to_files>-ostdout:///parse-zEach cell of extracted table data is serialized as follows:row- 0-indexed table row numbercolumn- 0-indexed table column numbervalue- the value of the cell as astrtype- the implied pythontypeof the data foundc_header- the column headerr_header- the row headerexcel_RC- the RC reference from the spreadsheet (e.g. B10)sheet- the name of the sheetf_name- the name of the filesqlite3eparse uses thepeeweepackage for ORM and database integration.  Theinterfacesmodule contains anExcelParsemodel that provides data persistence and a common
interface.To create aSQLite3database
with your parsed Excel data, use the following command:$mkdir.files$eparse-f<path_to_files>-osqlite3:///parse-zThis command will automatically generate a unique database filename
using theuuidpython package in the.files/sub-directory
of the working directory.  You may need to create this directory
before running this command, as shown.You can also specify a path and filename of your choosing, like so:$mkdir.files$eparse-f<path_to_files>-osqlite3:///path/filename.dbparse-zpostgreseparse also supportspostgresqlintegrations. As mentioned above,
you will needpsycopg2installed forpostgresqlintegrations
to work. The eparseBaseDatabaseInterfaceabstracts the
implementation details, so you would use this interface the same
way you use the others, with the exception of the endpoint.To use apostgresqldatabase as the source and/or destination
of your data, you would supply an--inputand/or--outputendpoint to the tool as follows:$eparse-opostgres://user:password@host:port/db_name...Where details likeuser,host,portare provided to
you by your db administrator. eparse will create the necessary
table(s) and indexes for you when inserting data into the database.QueryOnce you have stored parsed data, you can begin to query it using thepeeweeORM.  This can be done with the tool or directly with
the database.For example, query distinct column header names from a generatedSQLite3database as follows:$eparse-isqlite3:///.files/<db_file>-ostdout:///query-mget_c_headerc_headerTotalRowsDataTypesDistinctValues0ABC-col1502761DEF-col39813152GHI-col2012..............This command will give descriptive information of each distinct c_header
found, including total rows, unique data types, and distinct values.You can also get raw un-truncated data as follows:$eparse-t-isqlite3:///.files/<db_file>-ostdout:///queryFiltering data on content is easy.  Use the--filteroption as
follows:$eparse-isqlite3:///.files/<db_file>-ostdout:///query--filterf_name"somefile.xlsx"The above command will filter all rows from an Excel file namedsomefile.xlsx. You can use any of the followingdjango-style
filters:__eqequals X__ltless than X__lteless than or equal to X__gtgreater than X__gtegreater than or equal to X__nenot equal to X__inX is in__isis X__likelike expression, such as%somestr%, case sensitive__ilikelike expression, such as%somestr%, case insensitive__regexpregular expression matching such as^.*?foo.*?$Filters are applied to the ORM fields like so:--filterrow__gte 4all extracted table rows>= 5--filterf_name__ilike "%foo%"all data from filenames withfoo--filtervalue__ne 100all data with values other than100Queried data can even be stored into a new database for creating
curated data subsets, as follows:$eparse-isqlite3:///.files/<db_file>\-osqlite3:///.files/<subq_db_file>\query--filterf_name"somefile.xlsx"Since database files the tool generates when usingsqlite3:///areSQLitenative, you can also useSQLitedatabase client tools
and execute raw SQL like so:$sudoapt-getinstall-ysqlite3-tools$sqlite3.files/<db_file>SQLiteversion3.37.22022-01-0613:25:41Enter".help"forusagehints.sqlite>.schemaCREATETABLEIFNOTEXISTS"excelparse"("id"INTEGERNOTNULLPRIMARYKEY,"row"INTEGERNOTNULL,"column"INTEGERNOTNULL,"value"VARCHAR(255)NOTNULL,"type"VARCHAR(255)NOTNULL,"c_header"VARCHAR(255)NOTNULL,"r_header"VARCHAR(255)NOTNULL,"excel_RC"VARCHAR(255)NOTNULL,"name"VARCHAR(255)NOTNULL,"sheet"VARCHAR(255)NOTNULL,"f_name"VARCHAR(255)NOTNULL);sqlite>.headeronsqlite>SELECT*FROMexcelparseLIMIT1;id|row|column|value|type|c_header|r_header|excel_RC|name|sheet|f_name1|0|0|ABC|<class'str'>|SomeCol|SomeRow|B2|MyTable|Sheet1|myfile.xlsmMigrateeparse wouldn’t be a solid tool without the ability to migrate your
eparse databases for future code changes.  You can apply migrations
that ship with future versions of eparse as follows:$eparse-isqlite3:///.files/<db_file>migrate-m<migration>applied<migration>It is up to you to determine the migrations you need based on the
eparse version you are upgrading from and to. Migrations can be
found ineparse/migrations.pyUnstructuredIf you would like to use eparse to partition xls[x] files alongside unstructured, you can do so with our contributedpartitionandpartition_xlsxmodules. Simply import thepartitionfunction fromeparse.contrib.unstructured.partitionand use it instead ofpartitionfromunstructured.partition.autolike so:fromeparse.contrib.unstructured.partitionimportpartitionelements=partition(filename='some_file.xlsx',eparse_mode='...')Valideparse_modesettings are available ineparse.contrib.unstructured.xlsx._eparse_modes.ContributingAs an open-source project, contributions are always welcome. Please seeContributingfor more information.Licenseeparse is licensed under theMIT License. See theLICENSEfile for more details.ContactThanks for your support of eparse. Feel free to contact me atcpappala@gmail.comor connect with me onGithub.History0.7.3 (2023-08-23)Updated partition and partition_xlsx for latest unstructured0.7.2 (2023-08-22)Relaxed unstructured dependency requirement for compatibility0.7.1 (2023-08-20)Added unstructured contrib to core packageUopdaed README with instructions0.7.0 (2023-08-18)Added html conversion functions and an html data interfaceAdded get_table_digest to core0.6.3 (2023-06-30)Updated get_df_from_file to accept filename or io object0.6.1 (2023-06-22)Added get_df_from_file helper functionAdded contrib with unstructured partition handler0.5.0 (2023-06-14)Optimized database interfaces0.4.0 (2023-06-14)Added postgresql interface0.3.0 (2023-06-14)Added inteface classesAdded migrations0.2.0 (2023-06-12)Added migrate commandAdded 0.1.2 to 0.2.0 migration0.1.2 (2023-06-07)Updated README0.1.1 (2023-06-06)Updated requirementsUpdated README0.1.0 (2023-06-06)First release on PyPI.