Package: tabulator
tabulator-pyA library for reading and writing tabular data (csv/xls/json/etc).[Important Notice]We have releasedFrictionless Framework. This framework is logical continuation oftabulatorthat was extended to be a complete data solution. The change in not breaking for the existing software so no actions are required. Please read theMigration Guidefromtabulatorto Frictionless Framework.we continue to bug-fixtabulator@1.xin thisrepositoryas well as it's available onPyPias it was beforeplease note thatfrictionless@3.xversion's API, we're working on at the moment, is not stablewe will releasefrictionless@4.xby the end of 2020 to be the first SemVer/stable versionFeaturesSupports most common tabular formats: CSV, XLS, ODS, JSON, Google Sheets, SQL, and others. See complete listbelow.Loads local and remote data: Supports HTTP, FTP and S3.Low memory usage: Only the current row is kept in memory, so you can
large datasets.Supports compressed files: Using ZIP or GZIP algorithms.Extensible: You can add support for custom file formats and loaders (e.g.
FTP).ContentsGetting startedInstallationRunning on CLIRunning on PythonDocumentationWorking with StreamSupported schemesSupported file formatsCustom file sources and formatsAPI ReferencecliStreamLoaderParserWritervalidateTabulatorExceptionSourceErrorSchemeErrorFormatErrorEncodingErrorCompressionErrorIOErrorLoadingErrorHTTPErrorContributingChangelogGetting startedInstallation$pipinstalltabulatorRunning on CLITabulator ships with a simple CLI calledtabulatorto read tabular data. For
example:$tabulatorhttps://github.com/frictionlessdata/tabulator-py/raw/4c1b3943ac98be87b551d87a777d0f7ca4904701/data/table.csv.gz
id,name1,english2,中国人You can see all supported options by runningtabulator --help.Running on PythonfromtabulatorimportStreamwithStream('data.csv',headers=1)asstream:stream.headers# [header1, header2, ..]forrowinstream:print(row)# [value1, value2, ..]You can find other examples in theexamplesdirectory.DocumentationIn the following sections, we'll walk through some usage examples of
this library. All examples were tested with Python 3.6, but should
run fine with Python 3.3+.Working with StreamTheStreamclass represents a tabular stream. It takes the file path as thesourceargument. For example:<scheme>://path/to/file.<format>It uses this path to determine the file format (e.g. CSV or XLS) and scheme
(e.g. HTTP or postgresql). It also supports format extraction from URLs likehttp://example.com?format=csv. If necessary, you also can define these explicitly.Let's try it out. First, we create aStreamobject passing the path to a CSV file.importtabulatorstream=tabulator.Stream('data.csv')At this point, the file haven't been read yet. Let's open the stream so we can
read the contents.try:stream.open()excepttabulator.TabulatorExceptionase:pass# Handle exceptionThis will open the underlying data stream, read a small sample to detect the
file encoding, and prepare the data to be read. We catchtabulator.TabulatorExceptionhere, in case something goes wrong.We can now read the file contents. To iterate over each row, we do:forrowinstream.iter():print(row)# [value1, value2, ...]Thestream.iter()method will return each row data as a list of values. If
you prefer, you could callstream.iter(keyed=True)instead, which returns a
dictionary with the column names as keys. Either way, this method keeps only a
single row in memory at a time. This means it can handle handle large files
without consuming too much memory.If you want to read the entire file, usestream.read(). It accepts the same
arguments asstream.iter(), but returns all rows at once.stream.reset()rows=stream.read()Notice that we calledstream.reset()before reading the rows. This is because
internally, tabulator only keeps a pointer to its current location in the file.
If we didn't reset this pointer, we would read starting from where we stopped.
For example, if we ranstream.read()again, we would get an empty list, as
the internal file pointer is at the end of the file (because we've already read
it all). Depending on the file location, it might be necessary to download the
file again to rewind (e.g. when the file was loaded from the web).After we're done, close the stream with:stream.close()The entire example looks like:importtabulatorstream=tabulator.Stream('data.csv')try:stream.open()excepttabulator.TabulatorExceptionase:pass# Handle exceptionforrowinstream.iter():print(row)# [value1, value2, ...]stream.reset()# Rewind internal file pointerrows=stream.read()stream.close()It could be rewritten to use Python's context manager interface as:importtabulatortry:withtabulator.Stream('data.csv')asstream:forrowinstream.iter():print(row)stream.reset()rows=stream.read()excepttabulator.TabulatorExceptionase:passThis is the preferred way, as Python closes the stream automatically, even if some exception was thrown along the way.The full API documentation is available as docstrings in theStream source code.HeadersBy default, tabulator considers that all file rows are values (i.e. there is no
header).withStream([['name','age'],['Alex',21]])asstream:stream.headers# Nonestream.read()# [['name', 'age'], ['Alex', 21]]If you have a header row, you can use theheadersargument with the its row
number (starting from 1).# IntegerwithStream([['name','age'],['Alex',21]],headers=1)asstream:stream.headers# ['name', 'age']stream.read()# [['Alex', 21]]You can also pass a lists of strings to define the headers explicitly:withStream([['Alex',21]],headers=['name','age'])asstream:stream.headers# ['name', 'age']stream.read()# [['Alex', 21]]Tabulator also supports multiline headers for thexlsandxlsxformats.withStream('data.xlsx',headers=[1,3],fill_merged_cells=True)asstream:stream.headers# ['header from row 1-3']stream.read()# [['value1', 'value2', 'value3']]EncodingYou can specify the file encoding (e.g.utf-8andlatin1) via theencodingargument.withStream(source,encoding='latin1')asstream:stream.read()If this argument isn't set, Tabulator will try to infer it from the data. If you
get aUnicodeDecodeErrorwhile loading a file, try setting the encoding toutf-8.Compression (Python3-only)Tabulator supports both ZIP and GZIP compression methods. By default it'll infer from the file name:withStream('http://example.com/data.csv.zip')asstream:stream.read()You can also set it explicitly:withStream('data.csv.ext',compression='gz')asstream:stream.read()Optionsfilename: filename in zip file to process (default is first file)Allow htmlTheStreamclass raisestabulator.exceptions.FormatErrorif it detects HTML
contents. This helps avoiding the relatively common mistake of trying to load a
CSV file inside an HTML page, for example on GitHub.You can disable this behaviour using theallow_htmloption:withStream(source_with_html,allow_html=True)asstream:stream.read()# no exception on openSample sizeTo detect the file's headers, and run other checks like validating that the file
doesn't contain HTML, Tabulator reads a sample of rows on thestream.open()method. This data is available via thestream.sampleproperty. The number of
rows used can be defined via thesample_sizeparameters (defaults to 100).withStream(two_rows_source,sample_size=1)asstream:stream.sample# only first rowstream.read()# first and second rowsYou can disable this by settingsample_sizeto zero. This way, no data will be
read onstream.open().Bytes sample sizeTabulator needs to read a part of the file to infer its encoding. Thebytes_sample_sizearguments controls how many bytes will be read for this
detection (defaults to 10000).source='data/special/latin1.csv'withStream(source)asstream:stream.encoding# 'iso8859-2'You can disable this by settingbytes_sample_sizeto zero, in which case it'll
use the machine locale's default encoding.Ignore blank headersWhenTrue, tabulator will ignore columns that have blank headers (defaults toFalse).# Default behavioursource='text://header1,,header3\nvalue1,value2,value3'withStream(source,format='csv',headers=1)asstream:stream.headers# ['header1', '', 'header3']stream.read(keyed=True)# {'header1': 'value1', '': 'value2', 'header3': 'value3'}# Ignoring columns with blank headerssource='text://header1,,header3\nvalue1,value2,value3'withStream(source,format='csv',headers=1,ignore_blank_headers=True)asstream:stream.headers# ['header1', 'header3']stream.read(keyed=True)# {'header1': 'value1', 'header3': 'value3'}Ignore listed/not-listed headersThe option is similar to theignore_blank_headers. It removes arbitrary columns from the data based on the corresponding column names:# Ignore listed headers (omit columns)source='text://header1,header2,header3\nvalue1,value2,value3'withStream(source,format='csv',headers=1,ignore_listed_headers=['header2'])asstream:assertstream.headers==['header1','header3']assertstream.read(keyed=True)==[{'header1':'value1','header3':'value3'},]# Ignore NOT listed headers (pick colums)source='text://header1,header2,header3\nvalue1,value2,value3'withStream(source,format='csv',headers=1,ignore_not_listed_headers=['header2'])asstream:assertstream.headers==['header2']assertstream.read(keyed=True)==[{'header2':'value2'},]Force stringsWhenTrue, all rows' values will be converted to strings (defaults toFalse).Nonevalues will be converted to empty strings.# Default behaviourwithStream([['string',1,datetime.datetime(2017,12,1,17,00)]])asstream:stream.read()# [['string', 1, datetime.dateime(2017, 12, 1, 17, 00)]]# Forcing rows' values as stringswithStream([['string',1]],force_strings=True)asstream:stream.read()# [['string', '1', '2017-12-01 17:00:00']]Force parseWhenTrue, don't raise an exception when parsing a malformed row, but simply
return an empty row. Otherwise, tabulator raisestabulator.exceptions.SourceErrorwhen a row can't be parsed. Defaults toFalse.# Default behaviourwithStream([[1],'bad',[3]])asstream:stream.read()# raises tabulator.exceptions.SourceError# With force_parsewithStream([[1],'bad',[3]],force_parse=True)asstream:stream.read()# [[1], [], [3]]Skip rowsList of row numbers and/or strings to skip.
If it's a string, all rows that begin with it will be skipped (e.g. '#' and '//').
If it's the empty string, all rows that begin with an empty column will be skipped.source=[['John',1],['Alex',2],['#Sam',3],['Mike',4],['John',5]]withStream(source,skip_rows=[1,2,-1,'#'])asstream:stream.read()# [['Mike', 4]]If theheadersparameter is also set to be an integer, it will use the first not skipped row as a headers.source=[['#comment'],['name','order'],['John',1],['Alex',2]]withStream(source,headers=1,skip_rows=['#'])asstream:stream.headers# [['name', 'order']]stream.read()# [['Jogn', 1], ['Alex', 2]]Post parseList of functions that can filter or transform rows after they are parsed. These
functions receive theextended_rowscontaining the row's number, headers
list, and the row values list. They then process the rows, and yield or discard
them, modified or not.defskip_odd_rows(extended_rows):forrow_number,headers,rowinextended_rows:ifnotrow_number%2:yield(row_number,headers,row)defmultiply_by_two(extended_rows):forrow_number,headers,rowinextended_rows:doubled_row=list(map(lambdavalue:value*2,row))yield(row_number,headers,doubled_row)rows=[[1],[2],[3],[4],]withStream(rows,post_parse=[skip_odd_rows,multiply_by_two])asstream:stream.read()# [[4], [8]]These functions are applied in order, as a simple data pipeline. In the example
above,multiply_by_twojust sees the rows yielded byskip_odd_rows.Keyed and extended rowsThe methodsstream.iter()andstream.read()accept thekeyedandextendedflag arguments to modify how the rows are returned.By default, every row is returned as a list of its cells values:withStream([['name','age'],['Alex',21]])asstream:stream.read()# [['Alex', 21]]Withkeyed=True, the rows are returned as dictionaries, mapping the column names to their values in the row:withStream([['name','age'],['Alex',21]])asstream:stream.read(keyed=True)# [{'name': 'Alex', 'age': 21}]And withextended=True, the rows are returned as a tuple of(row_number, headers, row), thererow_numberis the current row number (starting from 1),headersis a list with the headers names, androwis a list with the rows
values:withStream([['name','age'],['Alex',21]])asstream:stream.read(extended=True)# (1, ['name', 'age'], ['Alex', 21])Supported schemess3It loads data from AWS S3. For private files you should provide credentials supported by theboto3library, for example, corresponding environment variables. Read more aboutconfiguringboto3.stream=Stream('s3://bucket/data.csv')Optionss3_endpoint_url- the endpoint URL to use. By default it'shttps://s3.amazonaws.com. For complex use cases, for example,goodtables's runs on a data package this option can be provided as an environment variableS3_ENDPOINT_URL.fileThe default scheme, a file in the local filesystem.stream=Stream('data.csv')http/https/ftp/ftpsIn Python 2,tabulatorcan't stream remote data sources because of a limitation in the underlying libraries. The whole data source will be loaded to the memory. In Python 3 there is no such problem and remote files are streamed.stream=Stream('https://example.com/data.csv')Optionshttp_session- arequests.Sessionobject. Read more in therequests docs.http_stream- Enables or disables HTTP streaming, when possible (enabled by default). Disable it if you'd like to preload the whole file into memory.http_timeout- This timeout will be used for arequestssession construction.streamThe source is a file-like Python object.withopen('data.csv')asfp:stream=Stream(fp)textThe source is a string containing the tabular data. Bothschemeandformatmust be set explicitly, as it's not possible to infer them.stream=Stream('name,age\nJohn, 21\n',scheme='text',format='csv')Supported file formatsIn this section, we'll describe the supported file formats, and their respective
configuration options and operations. Some formats only support read operations,
while others support both reading and writing.csv (read & write)stream=Stream('data.csv',delimiter=',')OptionsIt supports all options from the Python CSV library. Checktheir
documentationfor more information.xls/xlsx (read & write)Tabulator is unable to streamxlsfiles, so the entire file is loaded in
memory. Streaming is supported forxlsxfiles.stream=Stream('data.xls',sheet=1)Optionssheet: Sheet name or number (starting from 1).workbook_cache: An empty dictionary to handle workbook caching. If provided,tabulatorwill fill the dictionary withsource: tmpfile_pathpairs for remote workbooks. Each workbook will be downloaded only once and all the temporary files will be deleted on the process exit. Defauts: Nonefill_merged_cells: ifTrueit will unmerge and fill all merged cells by
a visible value. With this option enabled the parser can't stream data and
load the whole document into memory.preserve_formatting: ifTrueit will try to preserve text formatting of numeric and temporal cells returning it as strings according to how it looks in a spreadsheet (EXPERIMETAL)adjust_floating_point_error: ifTrueit will correct the Excel behaviour regarding floating point numbersods (read only)This format is not included to package by default. To use it please installtabulatorwith anodsextras:$ pip install tabulator[ods]Source should be a valid Open Office document.stream=Stream('data.ods',sheet=1)Optionssheet: Sheet name or number (starting from 1)gsheet (read only)A publicly-accessible Google Spreadsheet.stream=Stream('https://docs.google.com/spreadsheets/d/<id>?usp=sharing')stream=Stream('https://docs.google.com/spreadsheets/d/<id>edit#gid=<gid>')sql (read & write)Any database URL supported bysqlalchemy.stream=Stream('postgresql://name:pass@host:5432/database',table='data')Optionstable (required): Database table nameorder_by: SQL expression for row ordering (e.g.name DESC)Data Package (read only)This format is not included to package by default. You can enable it by
installing tabulator usingpip install tabulator[datapackage].ATabular Data Package.stream=Stream('datapackage.json',resource=1)Optionsresource: Resource name or index (starting from 0)inline (read only)Either a list of lists, or a list of dicts mapping the column names to their
respective values.stream=Stream([['name','age'],['John',21],['Alex',33]])stream=Stream([{'name':'John','age':21},{'name':'Alex','age':33}])json (read & write)JSON document containing a list of lists, or a list of dicts mapping the column
names to their respective values (see theinlineformat for an example).stream=Stream('data.json',property='key1.key2')Optionsproperty: JSON Path to the property containing the tabular data. For example, considering the JSON{"response": {"data": [...]}}, thepropertyshould be set toresponse.data.keyed(write): Save as array of arrays (default) or as array of dicts (keyed).ndjson (read only)stream=Stream('data.ndjson')tsv (read only)stream=Stream('data.tsv')html (read only)This format is not included to package by default. To use it please installtabulatorwith thehtmlextra:$ pip install tabulator[html]An HTML table element residing inside an HTML document.Supports simple tables (no merged cells) with any legal combination of the td, th, tbody & thead elements.Usuallyforamt='html'would need to be specified explicitly as web URLs don't always use the.htmlextension.stream=Stream('http://example.com/some/page.aspx',format='html'selector='.content .data table#id1',raw_html=True)Optionsselector: CSS selector for specifying whichtableelement to extract. By default it'stable, which takes the firsttableelement in the document. If empty, will assume the entire page is the table to be extracted (useful with some Excel formats).raw_html: False (default) to extract the textual contents of each cell. True to return the inner html without modification.Custom file sources and formatsTabulator is written with extensibility in mind, allowing you to add support for
new tabular file formats, schemes (e.g. ssh), and writers (e.g. MongoDB). There
are three components that allow this:LoadersLoads a stream from some location (e.g. ssh)ParsersParses a stream of tabular data in some format (e.g. xls)WritersWrites tabular data to some destination (e.g. MongoDB)In this section, we'll see how to write custom classes to extend any of these components.Custom loadersYou can add support for a new scheme (e.g. ssh) by creating a custom loader.
Custom loaders are implemented by inheriting from theLoaderclass, and
implementing its methods. This loader can then be used byStreamto load data
by passing it via thecustom_loaders={'scheme': CustomLoader}argument.The skeleton of a custom loader looks like:fromtabulatorimportLoaderclassCustomLoader(Loader):options=[]def__init__(self,bytes_sample_size,**options):passdefload(self,source,mode='t',encoding=None):# load logicwithStream(source,custom_loaders={'custom':CustomLoader})asstream:stream.read()You can see examples of how the loaders are implemented by looking in thetabulator.loadersmodule.Custom parsersYou can add support for a new file format by creating a custom parser. Similarly
to custom loaders, custom parsers are implemented by inheriting from theParserclass, and implementing its methods. This parser can then be used byStreamto parse data by passing it via thecustom_parsers={'format': CustomParser}argument.The skeleton of a custom parser looks like:fromtabulatorimportParserclassCustomParser(Parser):options=[]def__init__(self,loader,force_parse,**options):self.__loader=loaderdefopen(self,source,encoding=None):# open logicdefclose(self):# close logicdefreset(self):# reset logic@propertydefclosed(self):returnFalse@propertydefextended_rows(self):# extended rows logicwithStream(source,custom_parsers={'custom':CustomParser})asstream:stream.read()You can see examples of how parsers are implemented by looking in thetabulator.parsersmodule.Custom writersYou can add support to write files in a specific format by creating a custom
writer. The custom writers are implemented by inheriting from the baseWriterclass, and implementing its methods. This writer can then be used byStreamto
write data via thecustom_writers={'format': CustomWriter}argument.The skeleton of a custom writer looks like:fromtabulatorimportWriterclassCustomWriter(Writer):options=[]def__init__(self,**options):passdefwrite(self,source,target,headers=None,encoding=None):# write logicwithStream(source,custom_writers={'custom':CustomWriter})asstream:stream.save(target)You can see examples of how parsers are implemented by looking in thetabulator.writersmodule.API Referenceclicli(source,limit,**options)Command-line interfaceUsage: tabulator [OPTIONS] SOURCE

Options:
  --headers INTEGER
  --scheme TEXT
  --format TEXT
  --encoding TEXT
  --limit INTEGER
  --sheet TEXT/INTEGER (excel)
  --fill-merged-cells BOOLEAN (excel)
  --preserve-formatting BOOLEAN (excel)
  --adjust-floating-point-error BOOLEAN (excel)
  --table TEXT (sql)
  --order_by TEXT (sql)
  --resource TEXT/INTEGER (datapackage)
  --property TEXT (json)
  --keyed BOOLEAN (json)
  --version          Show the version and exit.
  --help             Show this message and exit.StreamStream(self,source,headers=None,scheme=None,format=None,encoding=None,compression=None,allow_html=False,sample_size=100,bytes_sample_size=10000,ignore_blank_headers=False,ignore_listed_headers=None,ignore_not_listed_headers=None,multiline_headers_joiner=' ',multiline_headers_duplicates=False,force_strings=False,force_parse=False,pick_rows=None,skip_rows=None,pick_fields=None,skip_fields=None,pick_columns=None,skip_columns=None,post_parse=[],custom_loaders={},custom_parsers={},custom_writers={},**options)Stream of tabular data.This is the maintabulatorclass. It loads a data source, and allows you
to stream its parsed contents.Argumentssource (str):
    Path to file as ``<scheme>://path/to/file.<format>``.
    If not explicitly set, the scheme (file, http, ...) and
    format (csv, xls, ...) are inferred from the source string.

headers (Union[int, List[int], List[str]], optional):
    Either a row
    number or list of row numbers (in case of multi-line headers) to be
    considered as headers (rows start counting at 1), or the actual
    headers defined a list of strings. If not set, all rows will be
    treated as containing values.

scheme (str, optional):
    Scheme for loading the file (file, http, ...).
    If not set, it'll be inferred from `source`.

format (str, optional):
    File source's format (csv, xls, ...). If not
    set, it'll be inferred from `source`. inferred

encoding (str, optional):
    Source encoding. If not set, it'll be inferred.

compression (str, optional):
    Source file compression (zip, ...). If not set, it'll be inferred.

pick_rows (List[Union[int, str, dict]], optional):
    The same as `skip_rows` but it's for picking rows instead of skipping.

skip_rows (List[Union[int, str, dict]], optional):
    List of row numbers, strings and regex patterns as dicts to skip.
    If a string, it'll skip rows that their first cells begin with it e.g. '#' and '//'.
    To skip only completely blank rows use `{'type': 'preset', 'value': 'blank'}`
    To provide a regex pattern use  `{'type': 'regex', 'value': '^#'}`
    For example: `skip_rows=[1, '# comment', {'type': 'regex', 'value': '^# (regex|comment)'}]`

pick_fields (str[]):
    When passed, ignores all columns with headers
    that the given list DOES NOT include

skip_fields (str[]):
    When passed, ignores all columns with headers
    that the given list includes. If it contains an empty string it will skip
    empty headers

sample_size (int, optional):
    Controls the number of sample rows used to
    infer properties from the data (headers, encoding, etc.). Set to
    ``0`` to disable sampling, in which case nothing will be inferred
    from the data. Defaults to ``config.DEFAULT_SAMPLE_SIZE``.

bytes_sample_size (int, optional):
    Same as `sample_size`, but instead
    of number of rows, controls number of bytes. Defaults to
    ``config.DEFAULT_BYTES_SAMPLE_SIZE``.

allow_html (bool, optional):
    Allow the file source to be an HTML page.
    If False, raises ``exceptions.FormatError`` if the loaded file is
    an HTML page. Defaults to False.

multiline_headers_joiner (str, optional):
    When passed, it's used to join multiline headers
    as `<passed-value>.join(header1_1, header1_2)`
    Defaults to ' ' (space).

multiline_headers_duplicates (bool, optional):
    By default tabulator will exclude a cell of a miltilne header from joining
    if it's exactly the same as the previous seen value in this field.
    Enabling this option will force duplicates inclusion
    Defaults to False.

force_strings (bool, optional):
    When True, casts all data to strings.
    Defaults to False.

force_parse (bool, optional):
    When True, don't raise exceptions when
    parsing malformed rows, simply returning an empty value. Defaults
    to False.

post_parse (List[function], optional):
    List of generator functions that
    receives a list of rows and headers, processes them, and yields
    them (or not). Useful to pre-process the data. Defaults to None.

custom_loaders (dict, optional):
    Dictionary with keys as scheme names,
    and values as their respective ``Loader`` class implementations.
    Defaults to None.

custom_parsers (dict, optional):
    Dictionary with keys as format names,
    and values as their respective ``Parser`` class implementations.
    Defaults to None.

custom_loaders (dict, optional):
    Dictionary with keys as writer format
    names, and values as their respective ``Writer`` class
    implementations. Defaults to None.

**options (Any, optional): Extra options passed to the loaders and parsers.stream.closedReturns True if the underlying stream is closed, False otherwise.Returnsbool: whether closedstream.compressionStream's compression ("no" if no compression)Returnsstr: compressionstream.encodingStream's encodingReturnsstr: encodingstream.formatPath's formatReturnsstr: formatstream.fragmentPath's fragmentReturnsstr: fragmentstream.hashReturns the SHA256 hash of the read chunks if availableReturnsstr/None: SHA256 hashstream.headersHeadersReturnsstr[]/None: headers if availablestream.sampleReturns the stream's rows used as sample.These sample rows are used internally to infer characteristics of the
source file (e.g. encoding, headers, ...).Returnslist[]: samplestream.schemePath's schemeReturnsstr: schemestream.sizeReturns the BYTE count of the read chunks if availableReturnsint/None: BYTE countstream.sourceSourceReturnsany: stream sourcestream.openstream.open()Opens the stream for reading.Raises:TabulatorException: if an errorstream.closestream.close()Closes the stream.stream.resetstream.reset()Resets the stream pointer to the beginning of the file.stream.iterstream.iter(keyed=False,extended=False)Iterate over the rows.Each row is returned in a format that depends on the argumentskeyedandextended. By default, each row is returned as list of their
values.Argumentskeyed (bool, optional):
When True, each returned row will be adictmapping the header name to its value in the current row.
For example,[{'name': 'J Smith', 'value': '10'}]. Ignored ifextendedis True. Defaults to False.extended (bool, optional):
When True, returns each row as a tuple
with row number (starts at 1), list of headers, and list of row
values. For example,(1, ['name', 'value'], ['J Smith', '10']).
Defaults to False.Raisesexceptions.TabulatorException: If the stream is closed.ReturnsIterator[Union[List[Any], Dict[str, Any], Tuple[int, List[str], List[Any]]]]:
The row itself. The format depends on the values ofkeyedandextendedarguments.stream.readstream.read(keyed=False,extended=False,limit=None)Returns a list of rows.Argumentskeyed (bool, optional): See :func:Stream.iter.extended (bool, optional): See :func:Stream.iter.limit (int, optional):
Number of rows to return. If None, returns all rows. Defaults to None.ReturnsList[Union[List[Any], Dict[str, Any], Tuple[int, List[str], List[Any]]]]:
The list of rows. The format depends on the values ofkeyedandextendedarguments.stream.savestream.save(target,format=None,encoding=None,**options)Save stream to the local filesystem.Argumentstarget (str): Path where to save the stream.format (str, optional):
The format the stream will be saved as. If
None, detects from thetargetpath. Defaults to None.encoding (str, optional):
Saved file encoding. Defaults toconfig.DEFAULT_ENCODING.**options: Extra options passed to the writer.Returnscount (int?): Written rows count if available
Building index...
Started generating documentation...LoaderLoader(self,bytes_sample_size,**options)Abstract class implemented by the data loadersThe loaders inherit and implement this class' methods to add support for a
new scheme (e.g. ssh).Argumentsbytes_sample_size (int): Sample size in bytes**options (dict): Loader optionsloader.optionsloader.loadloader.load(source,mode='t',encoding=None)Load source file.Argumentssource (str): Path to tabular source file.mode (str, optional):
Text stream mode,t(text) orb(binary).  Defaults tot.encoding (str, optional):
Source encoding. Auto-detect by default.ReturnsUnion[TextIO, BinaryIO]: I/O stream opened either as text or binary.ParserParser(self,loader,force_parse,**options)Abstract class implemented by the data parsers.The parsers inherit and implement this class' methods to add support for a
new file type.Argumentsloader (tabulator.Loader): Loader instance to read the file.force_parse (bool):
WhenTrue, the parser yields an empty extended
row tuple(row_number, None, [])when there is an error parsing a
row. Otherwise, it stops the iteration by raising the exceptiontabulator.exceptions.SourceError.**options (dict): Loader optionsparser.closedFlag telling if the parser is closed.Returnsbool: whether closedparser.encodingEncodingReturnsstr: encodingparser.extended_rowsReturns extended rows iterator.The extended rows are tuples containing(row_number, headers, row),RaisesSourceError:
Ifforce_parseisFalseand
a row can't be parsed, this exception will be raised.
Otherwise, an empty extended row is returned (i.e.(row_number, None, [])).Returns:
Iterator[Tuple[int, List[str], List[Any]]]:
Extended rows containing(row_number, headers, row), whereheadersis a list of the
header names (can beNone), androwis a list of row
values.parser.optionsparser.openparser.open(source,encoding=None)Open underlying file stream in the beginning of the file.The parser gets a byte or text stream from thetabulator.Loaderinstance and start emitting items.Argumentssource (str): Path to source table.encoding (str, optional): Source encoding. Auto-detect by default.ReturnsNoneparser.closeparser.close()Closes underlying file stream.parser.resetparser.reset()Resets underlying stream and current items list.Afterreset()is called, iterating over the items will start from the beginning.WriterWriter(self,**options)Abstract class implemented by the data writers.The writers inherit and implement this class' methods to add support for a
new file destination.Arguments**options (dict): Writer options.writer.optionswriter.writewriter.write(source,target,headers,encoding=None)Writes source data to target.Argumentssource (str): Source data.target (str): Write target.headers (List[str]): List of header names.encoding (str, optional): Source file encoding.Returnscount (int?): Written rows count if availablevalidatevalidate(source,scheme=None,format=None)Check if tabulator is able to load the source.Argumentssource (Union[str, IO]): The source path or IO object.scheme (str, optional): The source scheme. Auto-detect by default.format (str, optional): The source file format. Auto-detect by default.RaisesSchemeError: The file scheme is not supported.FormatError: The file format is not supported.Returnsbool: Whether tabulator is able to load the source file.TabulatorExceptionTabulatorException()Base class for all tabulator exceptions.SourceErrorSourceError()The source file could not be parsed correctly.SchemeErrorSchemeError()The file scheme is not supported.FormatErrorFormatError()The file format is unsupported or invalid.EncodingErrorEncodingError()Encoding errorCompressionErrorCompressionError()Compression errorIOErrorIOError()Local loading errorLoadingErrorLoadingError()Local loading errorHTTPErrorHTTPError()Remote loading errorContributingThe project follows theOpen Knowledge International coding standards.Recommended way to get started is to create and activate a project virtual environment.
To install package and development dependencies into active environment:$makeinstallTo run tests with linting and coverage:$maketestTo run tests without Internet:$ pytest -m 'not remoteChangelogHere described only breaking and the most important changes. The full changelog and documentation for all released versions could be found in nicely formattedcommit history.v1.53Add support for raw_html extraction in html parser (#341)v1.52Published stream.dialect (works only for csv, for now)v1.51Added experimental table discovery optionsv1.50Ensure that headers is always a list of stringsv1.49Addedworkbook_cacheargument for XLSX formatsv1.48Publishedstream.hashing_algorithmpropertyv1.47Addedhashing_algorithmparameterv1.46Fixed multiline headers mergingIntroduced amultiline_headers_duplicatesflagv1.45HTML format: adds support for empty selector (#321)v1.44Exposedstream.compressionv1.43Exposedstream.sourcev1.42Exposed format option to the CLIv1.41Implemented apick_rowsparameter (opposite toskip_rows)v1.40Implementedstream.save()returning count of written rowsv1.39Implemented JSON writer (#311)v1.38Usechardetfor encoding detection by default. Forcchardet:pip install tabulator[cchardet]. Due to a great deal of problems caused bycchartedfor non-Linux/Conda installations we're returning back to usingchardetby default.v1.37Raise IOError/HTTPError even a not-existent file has a bad format (#304)v1.36Implementedblankpreset forskip_rows(#302)v1.35Addedskip/pick_columnsaliases for (#293)v1.34Addedmultiline_headers_joinerargument (#291)v1.33Added support for regex patterns inskip_rows(#290)v1.32Added ability to skip columns (#293)v1.31AddedxlsxwriterAddedhtmlreaderv1.30Addedadjust_floating_point_errorparameter to thexlsxparserv1.29Implemented thestream.sizeandstream.hashpropertiesv1.28Added SQL writerv1.27Addedhttp_timeoutargument for thehttp/httpsformatv1.26Addedstream.fragmentfield showing e.g. Excel sheet's or DP resource's namev1.25Added support for thes3file scheme (data loading from AWS S3)v1.24Added support for compressed file-like objectsv1.23Added a setter for thestream.headerspropertyv1.22Theheadersparameter will now use the first not skipped row if theskip_rowsparameter is provided and there are comments on the top of a data source (see #264)v1.21Implemented experimentalpreserve_formattingfor xlsxv1.20Added support for specifying filename in zip sourcev1.19Updated behaviour:Forodsformat the boolean, integer and datetime native types are detected nowv1.18Updated behaviour:Forxlsformat the boolean, integer and datetime native types are detected nowv1.17Updated behaviour:Added support for Python 3.7v1.16New API added:skip_rowssupport for an empty string to skip rows with an empty first columnv1.15New API added:Format will be extracted from URLs likehttp://example.com?format=csvv1.14Updated behaviour:Nowxlsbooleans will be parsed as booleans not integersv1.13New API added:Theskip_rowsargument now supports negative numbers to skip rows starting from the endv1.12Updated behaviour:Instead of raising an exception, aUserWarningwarning will be emitted if an option isn't recognized.v1.11New API added:Addedhttp_sessionargument for thehttp/httpsformat (it usesrequestsnow)Added support for multiline headers:headersargument accept ranges like[1,3]v1.10New API added:Added support for compressed files i.e.zipandgzon Python3TheStreamconstructor now accepts acompressionargumentThehttp/httpsscheme now accepts ahttp_streamflagv1.9Improved behaviour:Theheadersargument allows to set the order for keyed sources and cherry-pick valuesv1.8New API added:FormatsXLS/XLSX/ODSsupports sheet names passed via thesheetargumentTheStreamconstructor accepts anignore_blank_headersoptionv1.7Improved behaviour:Rebaseddatapackageformat ondatapackage@1libraryv1.6New API added:Argumentsourcefor theStreamconstructor can be apathlib.Pathv1.5New API added:Argumentbytes_sample_sizefor theStreamconstructorv1.4Improved behaviour:Updated encoding name to a canonical formv1.3New API added:stream.schemestream.formatstream.encodingPromoted provisional API to stable API:Loader(custom loaders)Parser(custom parsers)Writer(custom writers)validatev1.2Improved behaviour:Autodetect common CSV delimitersv1.1New API added:Addedfill_merged_cellsoption toxls/xlsxformatsv1.0New API added:publishedLoader/Parser/WriterAPIAddedStreamargumentforce_stringsAddedStreamargumentforce_parseAddedStreamargumentcustom_writersDeprecated API removal:removedtopenandTable- useStreaminsteadremovedStreamargumentsloader/parser_options- use**optionsinsteadProvisional API changed:Updated theLoader/Parser/WriterAPI - please use an updated versionv0.15Provisional API added:Unofficial support forStreamargumentscustom_loaders/parsers