Repository: ipython/ipython

Issue:
Clear-all on save option
I sometimes forget to clear all before saving and checking the notebook into revision control, and it produces harder to read diffs and file sizes larger than I want to be commiting.  Being able to set an option within a notebook to only save in cleared format would be very useful.  Even nicer would be if it didn't clear the open notebook, just the saved version.



Discussion:
If you think about this the other way round (save output cells, True/False), it sounds pretty simple.

While we continue thinking about the best long-term solutions for VC integration, this could be an easy way to offer a good workflow right away that doesn't require any format changes or any decision on the more complex idea of splitting the nb into input/output files that we discussed a while ago.

We are planning on implementing a configuration panel soon and we can put this there.

Yes, that will be perfect.  We can discuss on-list a little bit what we want in it, what pieces of information visible in the panel will be available as worksheet metadata (so they persist across runs of the same notebook), etc.

I am not sure I understand how this would relate to the soon-to-appear configuration panel (or where that is being discussed), but I have hacked together a potential solution to this problem:

https://github.com/mforbes/ipython

If this is an appropriate direction, I can clean this up and submit a proper PR.  I have been trying this out for a while, and am discussing this here:

http://stackoverflow.com/questions/18734739/using-ipython-notebooks-under-version-control

I think this is slighlty too much restrictive. There should be another issue open about that, bu we would like a post-save hook that run any kind of conversion(s) with nbconvert afterward, so that you could,  each time you save, have a clean version but also html, pdf, ... etc potentially with custom templates. (nbconvert support ipynb->ipynb also)

Finally for the `clean` part, most of us woudl agree that this should be done with git `clean` and `smudge` filter, not by the application itself.

I totally agree with this being too restrictive, but could not find a way of adding a hook.  There was PR https://github.com/ipython/ipython/pull/3312 about hooks – closed as not "the direction we want to go" – and minimal discussion in issue https://github.com/ipython/ipython/issues/977.  I cannot find any further discussion about adding hooks: Is progress towards hooks being discussed somewhere?

The problem I have with relying on `clean` and `smudge` filters is that this means a separate solution is required for each version control system. In particular, I could not find a good way of doing this in mercurial.  To minimize maintenance, it would be best if the cleaning process could be kept VC agnostic.

The second issue (outstanding in all methods as far as I can see) is: how does one incorporate pulled "clean" changes into one's working notebook?  Probably something along the lines of first merging the two "clean" versions, then inserting the new content into the working notebook (removing any outdated output).  Any suggestions?  (Some way of updating one notebook from another seems to be required for this.  I am not sure if that is the jurisdiction of `nbconvert` or something else.)

Has there been any progress on this issue?

I've noticed that @minrk has posted scripts (https://gist.github.com/minrk/6176788 and https://gist.github.com/minrk/3719849; thank you for posting!). These are great in the short term, and their only disadvantage is kludging together a configuration that works, and like the original poster noted, remembering to clear all output before saving is something I'm bound to screw up if I don't script it. Is there any plan to bundle these scripts (or similar functionality in a command line script) as part of IPython?

I also noticed that there was some discussion of adding configuration options so that users could disable saving output. Last time I accessed http://ipython.org/ipython-doc/dev/config/options/notebook.html, I didn't see any options that would save only input cells. Is there still a plan to implement this feature?

There is no immediate plan to implement these as part of IPython itself.

I agree that the autosave / save as script (--script) would be most useful if they could be set at document level.

For example if I start up a ipynb server with the `--script` flag it'll save **all** files as .py. It is a very useful feature but in many cases I may not be interested in exporting all ipynb files to .py, but just those that contain common functions/workflows

This would be extremely useful for us, please consider adding just an optional flag!

Would be really nice if IPython supported this natively since the notebooks would become much more version control friendly.

Please implement a `post_save` hook soon, especially since `--script` support has been removed. Not all version control systems have smudge filters that can work as a universal replacement yet.


Resolving patch:
diff --git a/IPython/html/services/contents/filemanager.py b/IPython/html/services/contents/filemanager.py
--- a/IPython/html/services/contents/filemanager.py
+++ b/IPython/html/services/contents/filemanager.py
@@ -16,12 +16,37 @@
 from .manager import ContentsManager
 from IPython import nbformat
 from IPython.utils.io import atomic_writing
+from IPython.utils.importstring import import_item
 from IPython.utils.path import ensure_dir_exists
-from IPython.utils.traitlets import Unicode, Bool, TraitError
-from IPython.utils.py3compat import getcwd, str_to_unicode
+from IPython.utils.traitlets import Any, Unicode, Bool, TraitError
+from IPython.utils.py3compat import getcwd, str_to_unicode, string_types
 from IPython.utils import tz
 from IPython.html.utils import is_hidden, to_os_path, to_api_path
 
+_script_exporter = None
+
+def _post_save_script(model, os_path, contents_manager, **kwargs):
+    """convert notebooks to Python script after save with nbconvert
+
+    replaces `ipython notebook --script`
+    """
+    from IPython.nbconvert.exporters.script import ScriptExporter
+
+    if model['type'] != 'notebook':
+        return
+
+    global _script_exporter
+    if _script_exporter is None:
+        _script_exporter = ScriptExporter(parent=contents_manager)
+    log = contents_manager.log
+
+    base, ext = os.path.splitext(os_path)
+    py_fname = base + '.py'
+    script, resources = _script_exporter.from_filename(os_path)
+    script_fname = base + resources.get('output_extension', '.txt')
+    log.info("Saving script /%s", to_api_path(script_fname, contents_manager.root_dir))
+    with io.open(script_fname, 'w', encoding='utf-8') as f:
+        f.write(script)
 
 class FileContentsManager(ContentsManager):
 
@@ -64,13 +89,56 @@ def atomic_writing(self, os_path, *args, **kwargs):
             with atomic_writing(os_path, *args, **kwargs) as f:
                 yield f
     
-    save_script = Bool(False, config=True, help='DEPRECATED, IGNORED')
+    save_script = Bool(False, config=True, help='DEPRECATED, use post_save_hook')
     def _save_script_changed(self):
         self.log.warn("""
-        Automatically saving notebooks as scripts has been removed.
-        Use `ipython nbconvert --to python [notebook]` instead.
+        `--script` is deprecated. You can trigger nbconvert via pre- or post-save hooks:
+
+            ContentsManager.pre_save_hook
+            FileContentsManager.post_save_hook
+
+        A post-save hook has been registered that calls:
+
+            ipython nbconvert --to script [notebook]
+
+        which behaves similarly to `--script`.
         """)
 
+        self.post_save_hook = _post_save_script
+
+    post_save_hook = Any(None, config=True,
+        help="""Python callable or importstring thereof
+
+        to be called on the path of a file just saved.
+
+        This can be used to process the file on disk,
+        such as converting the notebook to a script or HTML via nbconvert.
+
+        It will be called as (all arguments passed by keyword):
+
+            hook(os_path=os_path, model=model, contents_manager=instance)
+
+        path: the filesystem path to the file just written
+        model: the model representing the file
+        contents_manager: this ContentsManager instance
+        """
+    )
+    def _post_save_hook_changed(self, name, old, new):
+        if new and isinstance(new, string_types):
+            self.post_save_hook = import_item(self.post_save_hook)
+        elif new:
+            if not callable(new):
+                raise TraitError("post_save_hook must be callable")
+
+    def run_post_save_hook(self, model, os_path):
+        """Run the post-save hook if defined, and log errors"""
+        if self.post_save_hook:
+            try:
+                self.log.debug("Running post-save hook on %s", os_path)
+                self.post_save_hook(os_path=os_path, model=model, contents_manager=self)
+            except Exception:
+                self.log.error("Post-save hook failed on %s", os_path, exc_info=True)
+
     def _root_dir_changed(self, name, old, new):
         """Do a bit of validation of the root_dir."""
         if not os.path.isabs(new):
@@ -404,6 +472,8 @@ def save(self, model, path=''):
         if 'content' not in model and model['type'] != 'directory':
             raise web.HTTPError(400, u'No file content provided')
 
+        self.run_pre_save_hook(model=model, path=path)
+
         # One checkpoint should always exist
         if self.file_exists(path) and not self.list_checkpoints(path):
             self.create_checkpoint(path)
@@ -433,6 +503,9 @@ def save(self, model, path=''):
         model = self.get(path, content=False)
         if validation_message:
             model['message'] = validation_message
+
+        self.run_post_save_hook(model=model, os_path=os_path)
+
         return model
 
     def update(self, model, path):
diff --git a/IPython/html/services/contents/manager.py b/IPython/html/services/contents/manager.py
--- a/IPython/html/services/contents/manager.py
+++ b/IPython/html/services/contents/manager.py
@@ -14,7 +14,9 @@
 from IPython.config.configurable import LoggingConfigurable
 from IPython.nbformat import sign, validate, ValidationError
 from IPython.nbformat.v4 import new_notebook
-from IPython.utils.traitlets import Instance, Unicode, List
+from IPython.utils.importstring import import_item
+from IPython.utils.traitlets import Instance, Unicode, List, Any, TraitError
+from IPython.utils.py3compat import string_types
 
 copy_pat = re.compile(r'\-Copy\d*\.')
 
@@ -60,6 +62,41 @@ def _notary_default(self):
         help="The base name used when creating untitled directories."
     )
 
+    pre_save_hook = Any(None, config=True,
+        help="""Python callable or importstring thereof
+
+        To be called on a contents model prior to save.
+
+        This can be used to process the structure,
+        such as removing notebook outputs or other side effects that
+        should not be saved.
+
+        It will be called as (all arguments passed by keyword):
+
+            hook(path=path, model=model, contents_manager=self)
+
+        model: the model to be saved. Includes file contents.
+               modifying this dict will affect the file that is stored.
+        path: the API path of the save destination
+        contents_manager: this ContentsManager instance
+        """
+    )
+    def _pre_save_hook_changed(self, name, old, new):
+        if new and isinstance(new, string_types):
+            self.pre_save_hook = import_item(self.pre_save_hook)
+        elif new:
+            if not callable(new):
+                raise TraitError("pre_save_hook must be callable")
+
+    def run_pre_save_hook(self, model, path, **kwargs):
+        """Run the pre-save hook if defined, and log errors"""
+        if self.pre_save_hook:
+            try:
+                self.log.debug("Running pre-save hook on %s", path)
+                self.pre_save_hook(model=model, path=path, contents_manager=self, **kwargs)
+            except Exception:
+                self.log.error("Pre-save hook failed on %s", path, exc_info=True)
+
     # ContentsManager API part 1: methods that must be
     # implemented in subclasses.
 
@@ -142,7 +179,11 @@ def get(self, path, content=True, type_=None, format=None):
         raise NotImplementedError('must be implemented in a subclass')
 
     def save(self, model, path):
-        """Save the file or directory and return the model with no content."""
+        """Save the file or directory and return the model with no content.
+
+        Save implementations should call self.run_pre_save_hook(model=model, path=path)
+        prior to writing any data.
+        """
         raise NotImplementedError('must be implemented in a subclass')
 
     def update(self, model, path):


Test patch: