Repository: pandas-dev/pandas

Issue:
ENH: Adding XlsxWriter as an ExcelWriter() option
related/closes #4517

Hi,

I am the author of the [XlsxWriter](https://github.com/jmcnamara/XlsxWriter) module, a Python module for writing Excel XLSX files.

I am interested in submitting a PR to add this as a option when writing Excel xlsx files via `to_excel()`.

It has a similar interface to the xlwt module but creates xlsx files. For large data files it is ~10x faster than OpenPyXL while maintaining low and constant memory usage. I occasionally see the current xlsx writer's speed and memory usage mentioned as a bug/limitation so it could be useful to have a fallback module option in those cases.

XlsxWriter also supports features such as charts and auto-filters that could be exposed at a later stage. See https://xlsxwriter.readthedocs.org/ for more details.

It uses standard libraries only and has an extensive test suite for Python 2.6, 2.7, 3.1, 3.2 and 3.3.

If you are interested I can work on a prototype and send a PR for you to evaluate. I have read CONTRIBUTING.md and I think I should be able to meet the criteria.

Regards,

John.



Discussion:
gr8!

could certainly be added as an optional deps see http://pandas.pydata.org/pandas-docs/dev/install.html#optional-dependencies; and could try in preference (if installed) over other writer libraries, then fallback to others (if installed)

you may need to rework the infrastructure a bit to do this,

e.g. at the top of the file various imports should be tried (see `io/html.py` for how this is done).

then the writers should take a list of `writers`? to try in sequence (else will finally raise if nothing is installed)

lmk if you need help

pls hook up travis as well; you will also need to add your library to `ci/requirements-???` for testing (you can add to all if you want), with a version specified (if you have multiple versions then you can do that too)

as a side issue; people have been thinking about how to pass formatting info to excel via these writers; either via some kind of dict or a `Style` type of object (generally per column, though could in theory be per cell), any thoughts on this?

> you may need to rework the infrastructure a bit to do this,

I'll try to make the changes as noninvasive as possible.

> pls hook up travis as well; you will also need to add your library to ci/requirements-???

Will do. XlsxWriter [already uses Travis](https://travis-ci.org/jmcnamara/XlsxWriter/builds) so it shouldn't be too painful. I normally test locally with a range of Python 2/3 versions as well.

John

gr8!

@jmcnamara how's this going? Definitely interested in being able to hook up other writers to the core...Does ExcelWriter have the same API as openpyxl for saving excel files?

@jtratner, @jreback 

> how's this going?

I made some initial progress getting XlsxWriter integrated and working, the code is on the [enh_xlsxwriter_dev](https://github.com/jmcnamara/pandas/tree/enh_xlsxwriter_dev) branch of my fork. For example the following now works using XlsxWriter:

``` python
    from pandas import ExcelWriter
    from pandas import DataFrame

    l1 = [1,2,3,4]
    l2 = [1,2,3,4]
    df1 = DataFrame({'Stimulus Time': l1, 'Reaction Time': l2})

    writer = ExcelWriter('path_to_file2.xlsx', writer_options={'use_xlsxwriter': True})
    df1.to_excel(writer, sheet_name='Sheet1', index=False)
    writer.save()
```

Dropping the optional `writer_options` parameter defaults back to OpenPyXL with similar results.

> Definitely interested in being able to hook up other writers to the core

I think it should be reasonably easy to plug in other writers as well. The required API changes would be minor and backward compatible: basically the additional dict of options for the underlying Excel writer (shown above).

I'll try to get it to a more complete state in the next week, with tests, so you review it properly and let me know if you think it will fit in with the current code.

> Does ExcelWriter have the same API as openpyxl for saving excel files?

It's API and scope is closer to Xlwt, i.e., it only creates new files and doesn't it doesn't rewrite files (although that could be simulated via the existing Pandas Excel reading functionality). In terms of functionality and features XlsxWriter goes beyond Xlwt or OpenPyXL and opens the possibility for [adding charts, images, autofilters and conditional formatting](https://xlsxwriter.readthedocs.org/en/latest/index.html) as Pandas Excel writer options.

However, I need to do a bit more integration work first. :-)

It should also be about 10x faster than OpenPyXL for large files. However, to get that performance the cells would have to be written in row order. Currently in `format.py` they are handled in column order via `chain`:

``` python
    def get_formatted_cells(self):
        for cell in itertools.chain(self._format_header(), self._format_body()
                                    ):
            cell.val = self._format_value(cell.val)
            yield cell
```

Would it be acceptable to convert this, and associated header handling functions, to row order via `izip_longest()` or some similar iterator?

Without it XlsxWriter should still be 5x faster than OpenPyXL but would consume a similar (large) amount of memory for large files. There in some background on this, if you are interested, in the [Working with Memory and Performance](https://xlsxwriter.readthedocs.org/en/latest/working_with_memory.html) section of the XlsxsWriter docs.

Anyway, I'll try push some more complete code for you and the others to look at in the next week.

John.

@jmcnamara I would like to see an API more like this

`df.to_excel(path_or_buf, sheet_name, engine='openpyxl|xlsxwriter'...., .....)`
additional options can be passed directly to the engine if needed (as kwargs)

and can default the engine (and possible use `get_option('io.excel.default_engine=....')`

the last bit is actually easy to set up see: https://github.com/pydata/pandas/blob/master/pandas/io/pytables.py#L170

@jmcnamara sounds good - it's definitely not a problem to go in row order vs. column order - I wouldn't know (without looking back at the history of the code) whether that was an explicit decision or not anyways. 5x speedup is absolutely non-trivial, so making a change to enable that is definitely a good thing.

@jreback 

I changed the API to `engine=`, added a default via the config handler and wrote some tests. The initial code is on my fork/branch.

In terms of the config it will probably need two items since the user may not be using the same engine for reading and writing. How about something like:

```
get_option('io.excel.writer_engine=....')
get_option('io.excel.reader_engine=....')
```

John

@jmcnamara pls make sure to hook up Travis
and you will need to update the ci/requirements file to make sure that the xlsxwriter is downloaded

@jmcnamara one thing that would be nice would be to, instead of hard coding the engine, just create the format like: `'_init_%s' % engine` and then check whether the engine exists on the class (so it's simple to bind new methods to the class and hook into the writer). The init method should take the path, so it could raise an error about getting an `.xls` file. (in fact, you could call like `_init_blah(path, **engine_kwargs)`).  I think it shouldn't require too much effort to run it that way (basically just extending the setup you already have).

You could do something like:

``` python
class _UnsupportedFileTypeError(Exception):
    pass

# __init__ method for ExcelWriter/

def __init__(self, path, engine=None, **engine_kwargs):

    if engine is None:
        engine = config.get_option('io.excel.engine')

    try:
        method = getattr(self, "_init_%s" % engine)
    except AttributeError:
        raise ValueError("No engine: %s" % engine)
    try:
        method(path, **engine_kwargs)
    except _UnsupportedFileTypeError:
        self._init_standard(path, **engine_kwargs)
```

and then the individual engines handle the rest of the setup. (like setting up the workbook, etc.).  If this seems too complicated, then we can always do it later after you've added in xlsxwriter and worked through the various issues with adding a new engine.

@jreback 

> make sure to hook up Travis and you will need to update the ci/requirements file to make sure that the xlsxwriter is downloaded

[Already done](https://travis-ci.org/jmcnamara/pandas) in both cases. :-)

@jmcnamara did you send this as a PR? (or its just on your branch atm?)

@jreback 

I haven't sent a PR. I am a few days work away from that as yet.

@jtratner 

> one thing that would be nice would be to, instead of hard coding the engine, just create the format like

That is a good suggestion. It makes it more generic and easier to add other engines. I also made the same change in another place.

I omitted the `_UnsupportedFileTypeError` part however since the `xlwt` and `openpyxl` called methods don't throw any exception.

@jreback @jtratner 

The code is getting close to a PR. I'll rebase/squash it onto another branch prior to that so that the merge is only one commit. I still have a bit more to add first but any comments on the existing code are welcome.

I wrote 28 `xlsxwriter` specific tests and added `xlsxwriter` variations of all of the existing `openpyxl` tests.

Here are some initial benchmark figures for the three supported writer modules:

```
$ python xltime.py 100
Engine = openpyxl   Rows =    100 time =   0.12
Engine = xlsxwriter Rows =    100 time =   0.02
Engine = xlwt       Rows =    100 time =   0.03

$ python xltime.py 1000
Engine = openpyxl   Rows =   1000 time =   0.61
Engine = xlsxwriter Rows =   1000 time =   0.14
Engine = xlwt       Rows =   1000 time =   0.12

$ python xltime.py 10000
Engine = openpyxl   Rows =  10000 time =   5.60
Engine = xlsxwriter Rows =  10000 time =   1.07
Engine = xlwt       Rows =  10000 time =   0.94

$ python xltime.py 50000
Engine = openpyxl   Rows =  50000 time =  26.80
Engine = xlsxwriter Rows =  50000 time =   4.89
Engine = xlwt       Rows =  50000 time =   4.40

$ python xltime.py 100000
Engine = openpyxl   Rows = 100000 time =  54.53
Engine = xlsxwriter Rows = 100000 time =  10.21

# xlwt and the xls format only supports 64,000 rows.
```

Here is the simple-minded benchmark program:

``` python

# Simple Benchmark test for the Pandas Excel writer modules.
import sys
from time import clock
from pandas import DataFrame
from pandas.core import config
import numpy as np


# Get the number of rows to test.
if len(sys.argv) > 1:
    row_max = int(sys.argv[1])
else:
    row_max = 1000


# Time the Excel writer modules.
def time_excel_engine(df, engine):

    filename = 'test_%s.xlsx' % engine
    config.set_option('io.excel.writer_engine', engine)

    start_time = clock()
    df.to_excel(filename, sheet_name='Sheet1', index=False)
    elapsed = clock() - start_time

    print("Engine = %-10s Rows = %6d time = %6.2f" % (engine, row_max, elapsed))


# Create a dataframe to write out to Excel.
df = DataFrame(np.random.randn(row_max, 4), columns=['A', 'B', 'C', 'D'])

# Time the modules.
time_excel_engine(df, 'openpyxl')

time_excel_engine(df, 'xlsxwriter')

if row_max < 64000:
    time_excel_engine(df, 'xlwt')

```

If I can refactor the format.py code to use row order then we should be able to get a doubling of speed from `xlsxwriter`. 

Adding the [PyExcelerate](https://github.com/kz26/PyExcelerate) module as a writer would get another 50% on that and with the reworked `engine` architecture it should be easy to add that or other modules.

you can put this in the form of a vbench

 I think you could try to create a vbench for this, nothing there now but create a file like  this: (named say excel): https://github.com/pydata/pandas/blob/master/vb_suite/ctors.py, and add it to the suite.py.

test using:

test_perf.sh -b <commit prior> -t <your last commit>

(lots of options to test_perf, e.g. you might want to start all of the tests with excel_, then you can add the 
-r excel option to ONLY test thoses specific vbenches


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -1356,7 +1356,7 @@ def to_csv(self, path_or_buf, sep=",", na_rep='', float_format=None,
                                      tupleize_cols=tupleize_cols)
         formatter.save()
 
-    def to_excel(self, excel_writer, sheet_name='sheet1', na_rep='',
+    def to_excel(self, excel_writer, sheet_name='Sheet1', na_rep='',
                  float_format=None, cols=None, header=True, index=True,
                  index_label=None, startrow=0, startcol=0, engine=None):
         """
@@ -1366,7 +1366,7 @@ def to_excel(self, excel_writer, sheet_name='sheet1', na_rep='',
         ----------
         excel_writer : string or ExcelWriter object
             File path or existing ExcelWriter
-        sheet_name : string, default 'sheet1'
+        sheet_name : string, default 'Sheet1'
             Name of sheet which will contain DataFrame
         na_rep : string, default ''
             Missing data representation
@@ -1397,8 +1397,8 @@ def to_excel(self, excel_writer, sheet_name='sheet1', na_rep='',
         to the existing workbook.  This can be used to save different
         DataFrames to one workbook
         >>> writer = ExcelWriter('output.xlsx')
-        >>> df1.to_excel(writer,'sheet1')
-        >>> df2.to_excel(writer,'sheet2')
+        >>> df1.to_excel(writer,'Sheet1')
+        >>> df2.to_excel(writer,'Sheet2')
         >>> writer.save()
         """
         from pandas.io.excel import ExcelWriter
diff --git a/pandas/io/excel.py b/pandas/io/excel.py
--- a/pandas/io/excel.py
+++ b/pandas/io/excel.py
@@ -596,6 +596,7 @@ def _convert_to_style(cls, style_dict, num_format_str=None):
         Parameters
         ----------
         style_dict: style dictionary to convert
+        num_format_str: optional number format string
         """
         import xlwt
 
@@ -611,3 +612,95 @@ def _convert_to_style(cls, style_dict, num_format_str=None):
 
 register_writer(_XlwtWriter)
 
+
+class _XlsxWriter(ExcelWriter):
+    engine = 'xlsxwriter'
+    supported_extensions = ('.xlsx',)
+
+    def __init__(self, path, **engine_kwargs):
+        # Use the xlsxwriter module as the Excel writer.
+        import xlsxwriter
+
+        super(_XlsxWriter, self).__init__(path, **engine_kwargs)
+
+        self.book = xlsxwriter.Workbook(path, **engine_kwargs)
+
+    def save(self):
+        """
+        Save workbook to disk.
+        """
+        return self.book.close()
+
+    def write_cells(self, cells, sheet_name=None, startrow=0, startcol=0):
+        # Write the frame cells using xlsxwriter.
+
+        sheet_name = self._get_sheet_name(sheet_name)
+
+        if sheet_name in self.sheets:
+            wks = self.sheets[sheet_name]
+        else:
+            wks = self.book.add_worksheet(sheet_name)
+            self.sheets[sheet_name] = wks
+
+        style_dict = {}
+
+        for cell in cells:
+            val = _conv_value(cell.val)
+
+            num_format_str = None
+            if isinstance(cell.val, datetime.datetime):
+                num_format_str = "YYYY-MM-DD HH:MM:SS"
+            if isinstance(cell.val, datetime.date):
+                num_format_str = "YYYY-MM-DD"
+
+            stylekey = json.dumps(cell.style)
+            if num_format_str:
+                stylekey += num_format_str
+
+            if stylekey in style_dict:
+                style = style_dict[stylekey]
+            else:
+                style = self._convert_to_style(cell.style, num_format_str)
+                style_dict[stylekey] = style
+
+            if cell.mergestart is not None and cell.mergeend is not None:
+                wks.merge_range(startrow + cell.row,
+                                startrow + cell.mergestart,
+                                startcol + cell.col,
+                                startcol + cell.mergeend,
+                                val, style)
+            else:
+                wks.write(startrow + cell.row,
+                          startcol + cell.col,
+                          val, style)
+
+    def _convert_to_style(self, style_dict, num_format_str=None):
+        """
+        converts a style_dict to an xlsxwriter format object
+        Parameters
+        ----------
+        style_dict: style dictionary to convert
+        num_format_str: optional number format string
+        """
+        if style_dict is None:
+            return None
+
+        # Create a XlsxWriter format object.
+        xl_format = self.book.add_format()
+
+        # Map the cell font to XlsxWriter font properties.
+        if style_dict.get('font'):
+            font = style_dict['font']
+            if font.get('bold'):
+                xl_format.set_bold()
+
+        # Map the cell borders to XlsxWriter border properties.
+        if style_dict.get('borders'):
+            xl_format.set_border()
+
+        if num_format_str is not None:
+            xl_format.set_num_format(num_format_str)
+
+        return xl_format
+
+register_writer(_XlsxWriter)
diff --git a/pandas/util/print_versions.py b/pandas/util/print_versions.py
--- a/pandas/util/print_versions.py
+++ b/pandas/util/print_versions.py
@@ -104,6 +104,12 @@ def show_versions():
     except:
         print("xlwt: Not installed")
 
+    try:
+        import xlsxwriter
+        print("xlsxwriter: %s" % xlsxwriter.__version__)
+    except:
+        print("xlsxwriter: Not installed")
+
     try:
         import sqlalchemy
         print("sqlalchemy: %s" % sqlalchemy.__version__)


Test patch:
diff --git a/pandas/io/tests/test_excel.py b/pandas/io/tests/test_excel.py
--- a/pandas/io/tests/test_excel.py
+++ b/pandas/io/tests/test_excel.py
@@ -16,9 +16,11 @@
     register_writer
 )
 from pandas.util.testing import ensure_clean
+from pandas.core.config import set_option, get_option
 import pandas.util.testing as tm
 import pandas as pd
 
+
 def _skip_if_no_xlrd():
     try:
         import xlrd
@@ -31,18 +33,25 @@ def _skip_if_no_xlrd():
 
 def _skip_if_no_xlwt():
     try:
-        import xlwt # NOQA
+        import xlwt  # NOQA
     except ImportError:
         raise nose.SkipTest('xlwt not installed, skipping')
 
 
 def _skip_if_no_openpyxl():
     try:
-        import openpyxl # NOQA
+        import openpyxl  # NOQA
     except ImportError:
         raise nose.SkipTest('openpyxl not installed, skipping')
 
 
+def _skip_if_no_xlsxwriter():
+    try:
+        import xlsxwriter  # NOQA
+    except ImportError:
+        raise nose.SkipTest('xlsxwriter not installed, skipping')
+
+
 def _skip_if_no_excelsuite():
     _skip_if_no_xlrd()
     _skip_if_no_xlwt()
@@ -268,15 +277,22 @@ def test_xlsx_table(self):
 
 
 class ExcelWriterBase(SharedItems):
-    # test cases to run with different extensions
-    # for each writer
-    # to add a writer test, define two things:
-    # 1. a check_skip function that skips your tests if your writer isn't
-    # installed
-    # 2. add a property ext, which is the file extension that your writer writes to
+    # Base class for test cases to run with different Excel writers.
+    # To add a writer test, define the following:
+    # 1. A check_skip function that skips your tests if your writer isn't
+    #    installed.
+    # 2. Add a property ext, which is the file extension that your writer
+    #    writes to.
+    # 3. Add a property engine_name, which is the name of the writer class.
     def setUp(self):
         self.check_skip()
         super(ExcelWriterBase, self).setUp()
+        self.option_name = 'io.excel.%s.writer' % self.ext
+        self.prev_engine = get_option(self.option_name)
+        set_option(self.option_name, self.engine_name)
+
+    def tearDown(self):
+        set_option(self.option_name, self.prev_engine)
 
     def test_excel_sheet_by_name_raise(self):
         _skip_if_no_xlrd()
@@ -790,6 +806,7 @@ def roundtrip(df, header=True, parser_hdr=0):
 
 class OpenpyxlTests(ExcelWriterBase, unittest.TestCase):
     ext = 'xlsx'
+    engine_name = 'openpyxl'
     check_skip = staticmethod(_skip_if_no_openpyxl)
 
     def test_to_excel_styleconverter(self):
@@ -820,6 +837,7 @@ def test_to_excel_styleconverter(self):
 
 class XlwtTests(ExcelWriterBase, unittest.TestCase):
     ext = 'xls'
+    engine_name = 'xlwt'
     check_skip = staticmethod(_skip_if_no_xlwt)
 
     def test_to_excel_styleconverter(self):
@@ -841,6 +859,52 @@ def test_to_excel_styleconverter(self):
         self.assertEquals(xlwt.Borders.THIN, xls_style.borders.left)
         self.assertEquals(xlwt.Alignment.HORZ_CENTER, xls_style.alignment.horz)
 
+
+class XlsxWriterTests(ExcelWriterBase, unittest.TestCase):
+    ext = 'xlsx'
+    engine_name = 'xlsxwriter'
+    check_skip = staticmethod(_skip_if_no_xlsxwriter)
+
+    # Override test from the Superclass to use assertAlmostEqual on the
+    # floating point values read back in from the output XlsxWriter file.
+    def test_roundtrip_indexlabels(self):
+        _skip_if_no_xlrd()
+        ext = self.ext
+        path = '__tmp_to_excel_from_excel_indexlabels__.' + ext
+
+        with ensure_clean(path) as path:
+
+            self.frame['A'][:5] = nan
+
+            self.frame.to_excel(path, 'test1')
+            self.frame.to_excel(path, 'test1', cols=['A', 'B'])
+            self.frame.to_excel(path, 'test1', header=False)
+            self.frame.to_excel(path, 'test1', index=False)
+
+            # test index_label
+            frame = (DataFrame(np.random.randn(10, 2)) >= 0)
+            frame.to_excel(path, 'test1', index_label=['test'])
+            reader = ExcelFile(path)
+            recons = reader.parse('test1', index_col=0).astype(np.int64)
+            frame.index.names = ['test']
+            self.assertEqual(frame.index.names, recons.index.names)
+
+            frame = (DataFrame(np.random.randn(10, 2)) >= 0)
+            frame.to_excel(
+                path, 'test1', index_label=['test', 'dummy', 'dummy2'])
+            reader = ExcelFile(path)
+            recons = reader.parse('test1', index_col=0).astype(np.int64)
+            frame.index.names = ['test']
+            self.assertEqual(frame.index.names, recons.index.names)
+
+            frame = (DataFrame(np.random.randn(10, 2)) >= 0)
+            frame.to_excel(path, 'test1', index_label='test')
+            reader = ExcelFile(path)
+            recons = reader.parse('test1', index_col=0).astype(np.int64)
+            frame.index.names = ['test']
+            self.assertAlmostEqual(frame.index.names, recons.index.names)
+
+
 class ExcelWriterEngineTests(unittest.TestCase):
     def test_ExcelWriter_dispatch(self):
         with tm.assertRaisesRegexp(ValueError, 'No engine'):
diff --git a/pandas/tests/test_panel.py b/pandas/tests/test_panel.py
--- a/pandas/tests/test_panel.py
+++ b/pandas/tests/test_panel.py
@@ -1429,6 +1429,26 @@ def test_to_excel(self):
                     recdf = reader.parse(str(item), index_col=0)
                     assert_frame_equal(df, recdf)
 
+    def test_to_excel_xlsxwriter(self):
+        try:
+            import xlrd
+            import xlsxwriter
+            from pandas.io.excel import ExcelFile
+        except ImportError:
+            raise nose.SkipTest("Requires xlrd and xlsxwriter. Skipping test.")
+
+        path = '__tmp__.xlsx'
+        with ensure_clean(path) as path:
+            self.panel.to_excel(path, engine='xlsxwriter')
+            try:
+                reader = ExcelFile(path)
+            except ImportError:
+                raise nose.SkipTest
+
+            for item, df in compat.iteritems(self.panel):
+                recdf = reader.parse(str(item), index_col=0)
+                assert_frame_equal(df, recdf)
+
     def test_dropna(self):
         p = Panel(np.random.randn(4, 5, 6), major_axis=list('abcde'))
         p.ix[:, ['b', 'd'], 0] = np.nan