Package: rxls
RXLSShortcut from Read XLS[X|B]Reading both XLSX and XLSB files, fast and memory-safe, with Python, into PyArrow.DescriptionThis module provides one function:readfor reading both .xlsx and .xlsb files.Simple usage showcase:importpolarsasplimportpandasaspdfromrxls.readerimportreadpolars_df=pl.from_arrow(read(some_file,header=True))pandas_df=read(some_file,header=True).to_pandas()Advanced usage examples.Some file has no header:fromrxls.readerimportreadtable=read(some_file,header=0)# produce `Unnamed: 0` ... `Unnamed: N` columnstable=read(some_file,header=['a','b','c'])# produce table with columns `a`, `b`, `c`, and raise an exception, when column count != 3.Header is placed in unknown row (but we know, it is in top-30 and one cell of it contains wordtimestamp)fromrxls.readerimportreadtable=read(some_file,header=True,lookup_head='^timestamp$')# First row of header must contain cell with word `timestamp`Some rows in columntimestampcontains unwanted cells (with subscripts, totals, or something else), but columnrow_iddoesn't:fromrxls.readerimportreadtable=read(some_file,header=True,row_filters='^row_id$')# After `header`, will be only rows, that contains non-null value in column `row_id`Some rows in columntimestampcontains unwanted cells, columnrow_iddoesn't, but we will ensure, that table has also non-null columnclient:fromrxls.readerimportreadtable=read(some_file,header=True,row_filters=['^row_id$','^client$'])# After `header`, will be only rows, that contains non-null value in columns `row_id` AND `client`Some rows in a sheet contains unwanted cells, but we know, thatrow_idcolumn cells are not null, orclientcolumn cells are not null either:fromrxls.readerimportreadtable=read(some_file,header=True,row_filters=['^row_id$','^client$'],row_filters_strategy='or')# After `header`, will be only rows, that contains non-null value in columns `row_id` OR `client`For tracking rows events:fromrxls.readerimportreadfromtqdmimporttqdmwithtqdm(desc='Reading some file',unit=' rows')astq:table=read(some_file,header=True,row_filters=['^row_id$','^client$'],row_filters_strategy='or',row_callback=tq.update)Some algorithm explanations:RXLS uses only TWO datatypes for cells of excel files: NUMERIC and STRING. It is because all other datatypes, as I discovered, are juststyles, applied to them (namedrepresentationin RXLS).So, when MS Excel show us a date (or time, or timestamp, or duration interval), it's actually a floating-point number "under the hood". This number, when is between [0.0, 1.0), presentspart of day(0.5 = 12 hours, 0.125 = 3 hours and vice versa), which useful fortimeanddurationdatatypes, and, when bigger or equals to1, presentsdays since 1900 with part of day. Fordate- decimals will be zero. All temporal cells (date/time, datetime, timedelta) will be converted totimestamp[ms]duringpreparestep. This behaviour may be overriden withdtypesargument - after all, pyarrow'stimestamp[ms]datatype may be convert todate64ortime32['ms']instantly and without any problems.durationdatatype in Excel files, usually, not presented, and pyarrow cannot casttimestamp['ms']toduration['ms']. For such conversion, you can extract timestamp column and convert it as:importpyarrowaspacolumn_name='duration_column'table=read(some_file,header=True)table.set_column(next(ifori,xinenumerate(table.column_names)ifx==column_name),pa.field(column_name,pa.duration('ms')),table[column_name].combine_chunks().view(pa.duration('ms'))# original dtype is pyarrow.timestamp['ms'], which is binary equivalent for pyarrow.duration['ms'])Strings most common representation is 4-byte little-endian integer (inxlsb) or some number (inxlsx). This number is an index ofsharedstring in special part of Excel file (xl/sharedStrings.[xml|bin]). This special part will be scannedon eachreadcall, before all other steps.Keeping all this in mind, I've implemented classchunkfor keeping original dataas-is, until it should be accessed during final table creation. Chunks, actually, are column parts, and contains non-null values with same type and representation. Chunk may be ofNULLtype, but it contains justcountof null values between not-null chunks.Original data will be contained insidechunkas-is, intact untilpreparestep:numpy.NDArray[numpy.float64]- simple numeric values (with optionaltemporalflag)numpy.NDArray[numpy.uint32]- forXLSBRkNumber-formatted floating-poing or integer values.pyarrow.LargeStringArray- forXLSXandXLSBstrings, that are not inshared, but presented as-is.pyarrow.UInt64Array- forXLSXandXLSBshared strings indices.int- fornullchunks: count of null valuesRXLS can skip entire chunks (or their parts) beforepreparestep, when they should be skipped (seelookup_headandrow_filtersarguments descriptions) - they will never be accessed afterwards.RXLS performs two main steps for a sheet of Excel file:readingandprepare.Onreadingstep, no computations will be performed - it's just fillingxl_seriesobjects bychunks, keeping original data.Onpreparestep, unwanted chunks will be dropped, and others will bepreparedas:numpy.NDArray[numpy.uint32]->numpy.NDArray[numpy.float64]- when numeric chunk hasrknubmerflag.numpy.NDArray[numpy.float64]->numpy.NDArray[numpy.int64]- when numeric chunk hastemporalflag.count of days with decimal part since 1900 (Windows)->milliseconds since 1970-01-01 (UNIX timestamp)pyarrow.UInt64Array->pyarrow.LargeStringArray- shared strings indices to actual data fromshared.int->pyarrow.NullArrayorpyarrow.Array[T, pyarrow.Scalar[T]]if column has chunks of another common datatypeT.These operations were totally optimized (and may performs much faster withnumbaalong with (or without)tbb- seeDependenciessection).Also, aboutlookup_headandheaderwith height > 1.If header has more than 1 row, every column will fill it's topmost empty cells by use corresponding non-empty cell values from left neighbour. All other cells will be leave as is. After this, non-empty cells will be concatenated from top to down with', 'separator. These operations performs for each column from left to right.So, RXLS features are:Lazy and fast column-wise computationsSmart header searching and row filteringKeeping datatypes and resolve conflicts.Notes:At the moment, RXLS just skip empty column (which has no header, nor any data).Parameters:Positionalonly:file: path to the file, or BytesIO with file.sheet: index or name of sheet for reading (default:0)Positional:header:bool:Truewhen header is presented in the sheet and have one row heightint:0when no header is presented in the sheet,Nfor header ofNrows height.list[str]: when no header is presented, but we are know, what it should be.Keyword-onlydtypes: Specify datatypes for columns.Sequence[pyarrow.DataType]: when we know dtype for each non-empty column.dict[str, pyarrow.DataType]: when we will override dtype for some columns.pyarrow.DataType: when all columns must be of this dtype.None: for original dtypes of columns.skip_cols: Skip some columns by their0-based indices (A == 0). Performs onreadingstep.skip_rows: Skip some rows on the top of the file. Performs onreadingstep.skip_rows_after_header: Skip some rows afterheader. Performs onpreparestep.take_rows: Stop reading after this row (0-based). Performs onreadingstep.take_rows_non_empty: Leave empty rows in resulting table. Performs onreadingstep.lookup_head: Regular expression for smart-search of the first row ofheader, orcolumnindex, where first non-empty cell is the top-level cell ofheader.lookup_size: Count of rows to perform lookup when searching forheader. Note: RXLS will raise an exception, iflookup_headwith thislookup_sizeis failed.row_filters: Regular expression(s) for columns, which content determines empty and non-empty rows.row_filters_strategy: Boolean operator(s) for situations with two or more columns inrow_filters.float_precision. All numeric values in MS Excel arefloating-pointunder the hood, so, when rounding wholefloat64column to this precision gives equal result to just truncate decimals, this column will be converted toint64.datetime_formats: One or more formats, which may appears in columns withconflicts.conflict_resolve: When column contains two or more datatypes, this is aconflict. When conflict cannot be resolved, whole column will be convert toutf-8. Conflicts may be resolved as:no: All parts in columns withconflictswill be convert toutf8.temporal: Try to convert non-temporal parts of column with some temporal parts to temporal (float64->timestampandutf8->timestamp(using default formats (ISO 8601), or as specified indatetime_formats)numeric: Try to convert non-numeric parts of column with some numeric parts to numeric (utf8->float64).all: Use both strategies to resolve conflicts. When some parts of column is temporal, try to convert all other parts to temporal (also enable two-step string converting:utf8->float64->timestamp)utf8_type_infers:(WIP)When resulting column isutf-8and all non-null cells of it passes regular expression ofnumericvalues, convert it tofloat64(and, maybe, toint64after).null_values: Advanced list of values, that should be skipped onreadingstep (orcallablepredicate for them).row_callback: Any callable, which may be called without arguments on each row event. Useful for progress tracking.DependenciesRequired:pyarrow>=14.0.2numpy>=1.24.4recordclass>=0.21.1typing_extensions>=4.9.0Optional:numba>=0.58.1(increase import time, but reading speed also increases x3/x4 and up)tbb>=2021.11.0(only for numba - additional performance gain)polars>=0.20.4(if needs to parse timestamps with milliseconds/microseconds/nanoseconds or AM/PM with timezone)pandas>=2.0.3(for pyarrowto_pandasfunctionality)tqdm>=4.66.1(fast progress tracking)