Repository: pandas-dev/pandas

Issue:
Allow ExcelWriter() to add sheets to existing workbook
The ability of ExcelWriter to save different dataframes to different worksheets is great for sharing those dfs with the python-deficient.  But this quickly leads to a need to _add_ worksheets to an existing workbook, not just creating one from scratch; something like:

```
df0=pd.DataFrame(np.arange(3))
df0.to_excel('foo.xlsx','Data 0')

df1=pd.DataFrame(np.arange(2))
df1.to_excel('foo.xlsx','Data 1')
```

The following little diff to io/parsers.py implements this behavior for *.xlsx files:

```
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
index 89f892d..7f010ee 100644
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -2099,12 +2099,19 @@ class ExcelWriter(object):
             self.fm_date = xlwt.easyxf(num_format_str='YYYY-MM-DD')
         else:
             from openpyxl.workbook import Workbook
-            self.book = Workbook()  # optimized_write=True)
-            # open pyxl 1.6.1 adds a dummy sheet remove it
-            if self.book.worksheets:
-                self.book.remove_sheet(self.book.worksheets[0])
+            from openpyxl.reader.excel import load_workbook
+
+            try:
+               self.book=load_workbook(filename = path)
+               self.sheets={wks.title:wks for wks in self.book.worksheets}
+            except InvalidFileException:
+                self.book = Workbook()  # optimized_write=True)
+                # open pyxl 1.6.1 adds a dummy sheet remove it
+                if self.book.worksheets:
+                    self.book.remove_sheet(self.book.worksheets[0])
+                self.sheets = {}
+
         self.path = path
-        self.sheets = {}
         self.cur_sheet = None
```

Doing this for *.xls files is a little harder.



Discussion:
@jtratner is this still a bug/needed enhancement?

Because of how `to_excel` is set up, this would mean reading in and then writing the file each time (because `to_excel` with a path argument saves the file). The right way to do this is to use `ExcelWriter`:

``` python
import pandas as pd
writer = pd.ExcelWriter('foo.xlsx')
df.to_excel(writer, 'Data 0')
df.to_excel(writer, 'Data 1')
writer.save()
```

I could see (eventually) adding an option to ExcelWriter that doesn't overwrite the file. But, yet again, that may mean writing in the entire file first. I don't know.

I'm going to add something to the docs about this, maybe a test case with this, and I'll look into adding an option to read in the file, but it depends on how xlwt and openpyxl work.

@jtratner what about a context manager `get_excel`?

```
with get_excel('foo.xlsx') as e:
    df.to_excel(e,'Data 0)
    df.to_excel(e,'Data 1)
```

?

how about we just make ExcelWriter into a contextmanager instead?  it'll just call save at the end. Much simpler.

@ligon you can do this now this way:

``` python
with ExcelWriter('foo.xlsx') as writer:
    df.to_excel(writer, 'Data 0')
    df2.to_excel(writer, 'Data 1')
```

If you don't use the with statement, just have to call save() at the end.

Excellent.  And great that it has an **exit** method.

Thanks,
-Ethan Ligon

Ethan Ligon, Associate Professor
Agricultural & Resource Economics
University of California, Berkeley

I was extremely interesting by the request made by @ligon - but seems this is already there.
However using 0.12.0 pd version, when I am doing:
df = DataFrame([1,2,3])
df2 = DataFrame([5,5,6])
with ExcelWriter('foo.xlsx') as writer:
    df.to_excel(writer, 'Data 0')
    df2.to_excel(writer, 'Data 1')

Assumning foo.xlsx was containing a sheet named 'bar', basgot delete after the command run. While as per your comment, i was expecting to keep it in my foo excel file. Is that a bug?

is it hard to add sheets to an  existing excel file on the disk?
import pandas as pd 
import numpy as np
a=pd.DataFrame(np.random.random((3,1)))
excel_writer=pd.ExcelWriter('c:\excel.xlsx')
a.to_excel(excel_writer, 'a1')
excel_writer.save()

excel_writer=pd.ExcelWriter('c:\excel.xlsx')
a.to_excel(excel_writer, 'a2')
excel_writer.save()

here only sheet 'a2"  is save, but I like to save both 'a1' and 'a2'.

I know it is possible to add sheets to an existing workbook.

It's definitely possible to add sheets to an existing workbook, but it's
not necessarily easy to do it with pandas. I think you'd have to read the
workbook separately and then pass it into the ExcelWriter class... That
would be something we could consider supporting.

And I think if you subclass the ExcelWriter instance you want to use and
overwrite its__init__ method, as long as you set self.book it should work.
That said, no guarantee that this would continue to work in future
versions, since it's only a quasi-public API

[This stackoverflow workaround](http://stackoverflow.com/a/20221655/548792), which is based in `openpyxl`, ~~may work~~
(EDIT: indeed works, checked with `pandas-0.17.0`):

``` python
import pandas
from openpyxl import load_workbook

book = load_workbook('Masterfile.xlsx')
writer = pandas.ExcelWriter('Masterfile.xlsx', engine='openpyxl') 
writer.book = book
writer.sheets = dict((ws.title, ws) for ws in book.worksheets)

data_filtered.to_excel(writer, "Main", cols=['Diff1', 'Diff2'])

writer.save()
```

this would be pretty easy to implement inside ExcelWriter (Oder patch above)

prob just need to add a mode=kw and default to w and make a be append 

Was this ever patched?

it seems that you _can_ work around it (see above), but I suppose would be nice to actually do it from pandas.

Hi, any follow up on this issue?
I can provide a use case: I have excel files with pivot tables and pivot graphs that I need to reach out people not proficient in Python. 

My idea was to use pandas to add a sheet that contains the data for the pivot. But up to know I am stuck and the proposed workaround, thought not difficult, sounds a bit cumbersome . It would make sense to jsut have an option whether overwrite an existing file.
Let me echo @jreback , it would be **super** nice if I could just add a sheet to an excel workbook from pandas.
To be clear, we agree that this would be a nice functionality, and would certainly welcome a contribution from the community. 
Best way to have this in pandas is to make a PR!
@jmcnamara how to use pandas.to_excel(Writer maybe use pandas.ExcelWriter) to add some data to an existed file , but not rewrite it??
@aa3222119 That is exactly what this issue is about: an enhancement request to add this ability, which is not yet possible today. 

(BTW, it is not needed to post the same question in multiple issues)

sorry. i will delete that one . BTW, will that be possible some day later？ @jorisvandenbossche
> BTW will that be possible some day later？ @jmcnamara

This isn't and won't be possible when using XlsxWriter as the engine. It should be possible when using OpenPyXL. I've seen some examples on SO like this one: https://github.com/jmcnamara/excel-writer-xlsx/issues/157

  3Q very much!  @jmcnamara
it is exactly what you said .  use openpyxl :+1: 


import pandas as pd 
from openpyxl import load_workbook
book = load_workbook('text.xlsx')
writer = pd.ExcelWriter('text.xlsx', engine='openpyxl') 
writer.book = book
pd.DataFrame(userProfile,index=[1]).to_excel(writer,'sheet111',startrow=7,startcol=7)
pd.DataFrame(userProfile,index=[1]).to_excel(writer,'sheet123',startrow=0,startcol=0)
writer.save()
pd.DataFrame(userProfile,index=[1]).to_excel(writer,'sheet123',startrow=3,startcol=3)
writer.save()


all can be added to text.xlsx.
[https://github.com/pandas-dev/pandas/issues/3441](url)

[openpyxl allows you to put DataFrames wherever you want them](https://openpyxl.readthedocs.io/en/latest/pandas.html)
@ankostis, @aa3222119 when I follow the steps you comment, I always reach the following error:

Traceback (most recent call last):
  File "./name_manipulation.py", line 60, in <module>
    df.to_excel(excel_writer, 'iadatasheet', startcol=0, startrow=5, columns=['codes', 'Zona Basica de Salud', month+"-"+year], index=False)
  File "/Users/jgonzalez.iacs/Projects/SIIDI/PYTHON_ETLs/venv/lib/python3.4/site-packages/pandas/core/frame.py", line 1464, in to_excel
    startrow=startrow, startcol=startcol)
  File "/Users/jgonzalez.iacs/Projects/SIIDI/PYTHON_ETLs/venv/lib/python3.4/site-packages/pandas/io/excel.py", line 1306, in write_cells
    wks = self.book.create_sheet()
AttributeError: 'str' object has no attribute 'create_sheet'

So, there is not solution yet, right?

Thanks

Maybe the API has changed - it definitely worked back then.
@jgonzale which engine are you using?
@jgonzale by what python said , your excel_writer.book maybe just a str but not a workbook?
@aa3222119 Oh geez! You were right! Messing around with very similar names!

Thank you very much! 👏  👏 👏 


thank you very much!
Hello
I have some use case where it would be useful:
Even with the ExcelWriter trick as:

with ExcelWriter('foo.xlsx') as writer:
df.to_excel(writer, 'Data 0')
df2.to_excel(writer, 'Data 1')

you can't add a plot that you need without saving the file and reopening it. With the risk of meddling with any formatting you have in the workbook.

There is indeed the workaround to use the plotting functions from pandas to save these in the files, but (there is a but), when you need something a little more sophisticated like showing a PCA components graph you built from scikitlearn PCA and matplotlib, then it becomes tedious.

Hence
a pandas.nondf_manager (non df object or filename).to_excel(usual syntax)
would be exceedingly fine.
Thanks.
I don't know how it is possible, however, it works for me

```
create_excel = 0
if plot_spectra != 0:
    for x in range(min_sigma, max_sigma, step_size):
        # apply gaussian
        df1 = gaussian_filter(df, sigma=x, mode=padding_mode)
        df2 = pd.DataFrame(df1)
        if save_file:
            if save_csv:
                df2.to_csv('{} {}{}.csv'.format(Output_file, 'sigma_', x,))
            if save_xlsx:
                if os.path.isfile('{}.xlsx'.format(Output_file)):
                    print("Warning! Excel file is exist")
                    break
                if create_excel == 0:
                    xlsx_writer = pd.ExcelWriter('{}.xlsx'.format(Output_file), engine='xlsxwriter')                                          
                    create_excel += 1
                df2.to_excel(xlsx_writer, '{}{}'.format('sigma_', x))
                if x == max_sigma-1:
                    xlsx_writer.save()
```

At the end, I got the excel file which have several work sheets.
@orbitalz you are creating an excel file the first time (``xlsx_writer = pd.ExcelWriter(..)``), and then adding multiple sheets to that file object. That is supported, but this issue is about adding sheets to an *existing* excel file. 
I'm sorry for misunderstanding the topic and Thank you for pointing me out :)
@orbitalz  You solve my problem ,but I don't known how it works  

Resolving patch:
diff --git a/pandas/io/excel.py b/pandas/io/excel.py
--- a/pandas/io/excel.py
+++ b/pandas/io/excel.py
@@ -804,6 +804,10 @@ class ExcelWriter(object):
     datetime_format : string, default None
         Format string for datetime objects written into Excel files
         (e.g. 'YYYY-MM-DD HH:MM:SS')
+    mode : {'w' or 'a'}, default 'w'
+        File mode to use (write or append).
+
+    .. versionadded:: 0.24.0
 
     Notes
     -----
@@ -897,7 +901,8 @@ def save(self):
         pass
 
     def __init__(self, path, engine=None,
-                 date_format=None, datetime_format=None, **engine_kwargs):
+                 date_format=None, datetime_format=None, mode='w',
+                 **engine_kwargs):
         # validate that this engine can handle the extension
         if isinstance(path, string_types):
             ext = os.path.splitext(path)[-1]
@@ -919,6 +924,8 @@ def __init__(self, path, engine=None,
         else:
             self.datetime_format = datetime_format
 
+        self.mode = mode
+
     def __fspath__(self):
         return _stringify_path(self.path)
 
@@ -993,23 +1000,27 @@ class _OpenpyxlWriter(ExcelWriter):
     engine = 'openpyxl'
     supported_extensions = ('.xlsx', '.xlsm')
 
-    def __init__(self, path, engine=None, **engine_kwargs):
+    def __init__(self, path, engine=None, mode='w', **engine_kwargs):
         # Use the openpyxl module as the Excel writer.
         from openpyxl.workbook import Workbook
 
-        super(_OpenpyxlWriter, self).__init__(path, **engine_kwargs)
+        super(_OpenpyxlWriter, self).__init__(path, mode=mode, **engine_kwargs)
 
-        # Create workbook object with default optimized_write=True.
-        self.book = Workbook()
+        if self.mode == 'a':  # Load from existing workbook
+            from openpyxl import load_workbook
+            book = load_workbook(self.path)
+            self.book = book
+        else:
+            # Create workbook object with default optimized_write=True.
+            self.book = Workbook()
 
-        # Openpyxl 1.6.1 adds a dummy sheet. We remove it.
-        if self.book.worksheets:
-            try:
-                self.book.remove(self.book.worksheets[0])
-            except AttributeError:
+            if self.book.worksheets:
+                try:
+                    self.book.remove(self.book.worksheets[0])
+                except AttributeError:
 
-                # compat
-                self.book.remove_sheet(self.book.worksheets[0])
+                    # compat - for openpyxl <= 2.4
+                    self.book.remove_sheet(self.book.worksheets[0])
 
     def save(self):
         """
@@ -1443,11 +1454,16 @@ class _XlwtWriter(ExcelWriter):
     engine = 'xlwt'
     supported_extensions = ('.xls',)
 
-    def __init__(self, path, engine=None, encoding=None, **engine_kwargs):
+    def __init__(self, path, engine=None, encoding=None, mode='w',
+                 **engine_kwargs):
         # Use the xlwt module as the Excel writer.
         import xlwt
         engine_kwargs['engine'] = engine
-        super(_XlwtWriter, self).__init__(path, **engine_kwargs)
+
+        if mode == 'a':
+            raise ValueError('Append mode is not supported with xlwt!')
+
+        super(_XlwtWriter, self).__init__(path, mode=mode, **engine_kwargs)
 
         if encoding is None:
             encoding = 'ascii'
@@ -1713,13 +1729,18 @@ class _XlsxWriter(ExcelWriter):
     supported_extensions = ('.xlsx',)
 
     def __init__(self, path, engine=None,
-                 date_format=None, datetime_format=None, **engine_kwargs):
+                 date_format=None, datetime_format=None, mode='w',
+                 **engine_kwargs):
         # Use the xlsxwriter module as the Excel writer.
         import xlsxwriter
 
+        if mode == 'a':
+            raise ValueError('Append mode is not supported with xlsxwriter!')
+
         super(_XlsxWriter, self).__init__(path, engine=engine,
                                           date_format=date_format,
                                           datetime_format=datetime_format,
+                                          mode=mode,
                                           **engine_kwargs)
 
         self.book = xlsxwriter.Workbook(path, **engine_kwargs)


Test patch:
diff --git a/pandas/tests/io/test_excel.py b/pandas/tests/io/test_excel.py
--- a/pandas/tests/io/test_excel.py
+++ b/pandas/tests/io/test_excel.py
@@ -2006,6 +2006,31 @@ def test_write_cells_merge_styled(self, merge_cells, ext, engine):
             assert xcell_b1.font == openpyxl_sty_merged
             assert xcell_a2.font == openpyxl_sty_merged
 
+    @pytest.mark.parametrize("mode,expected", [
+        ('w', ['baz']), ('a', ['foo', 'bar', 'baz'])])
+    def test_write_append_mode(self, merge_cells, ext, engine, mode, expected):
+        import openpyxl
+        df = DataFrame([1], columns=['baz'])
+
+        with ensure_clean(ext) as f:
+            wb = openpyxl.Workbook()
+            wb.worksheets[0].title = 'foo'
+            wb.worksheets[0]['A1'].value = 'foo'
+            wb.create_sheet('bar')
+            wb.worksheets[1]['A1'].value = 'bar'
+            wb.save(f)
+
+            writer = ExcelWriter(f, engine=engine, mode=mode)
+            df.to_excel(writer, sheet_name='baz', index=False)
+            writer.save()
+
+            wb2 = openpyxl.load_workbook(f)
+            result = [sheet.title for sheet in wb2.worksheets]
+            assert result == expected
+
+            for index, cell_value in enumerate(expected):
+                assert wb2.worksheets[index]['A1'].value == cell_value
+
 
 @td.skip_if_no('xlwt')
 @pytest.mark.parametrize("merge_cells,ext,engine", [
@@ -2060,6 +2085,13 @@ def test_to_excel_styleconverter(self, merge_cells, ext, engine):
         assert xlwt.Alignment.HORZ_CENTER == xls_style.alignment.horz
         assert xlwt.Alignment.VERT_TOP == xls_style.alignment.vert
 
+    def test_write_append_mode_raises(self, merge_cells, ext, engine):
+        msg = "Append mode is not supported with xlwt!"
+
+        with ensure_clean(ext) as f:
+            with tm.assert_raises_regex(ValueError, msg):
+                ExcelWriter(f, engine=engine, mode='a')
+
 
 @td.skip_if_no('xlsxwriter')
 @pytest.mark.parametrize("merge_cells,ext,engine", [
@@ -2111,6 +2143,13 @@ def test_column_format(self, merge_cells, ext, engine):
 
             assert read_num_format == num_format
 
+    def test_write_append_mode_raises(self, merge_cells, ext, engine):
+        msg = "Append mode is not supported with xlsxwriter!"
+
+        with ensure_clean(ext) as f:
+            with tm.assert_raises_regex(ValueError, msg):
+                ExcelWriter(f, engine=engine, mode='a')
+
 
 class TestExcelWriterEngineTests(object):