internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getnikola/nikola

Issue:
Add Zopfli support
Only half-serious, but hey, 5% smaller!

http://code.google.com/p/zopfli/



Discussion:
How about support for pigz? It's a parallel gzip, useful for those with multi-core machines.

Killing 2 birds with 1 stone, pigz v2.3 adds support for zopfli (a new -11 flag): http://mail.zlib.net/pipermail/pigz-announce_zlib.net/2013-March/000007.html

At the moment, nikola does gzipping internally instead of using an external utility. What would be the best way to refactor it so it used an external utility instead?

Probably moving "create_gzipped_copy" into utils and make it use pigz via subprocess if it's available.

Looking at the code, a thought: how about turning file gzip'ing into a filter?

Filters are supposed to work in place, and we really want a gzipped copy...


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -140,6 +140,7 @@ def __init__(self, **config):
             'FILES_FOLDERS': {'files': ''},
             'FILTERS': {},
             'GALLERY_PATH': 'galleries',
+            'GZIP_COMMAND': None,
             'GZIP_FILES': False,
             'GZIP_EXTENSIONS': ('.txt', '.htm', '.html', '.css', '.js', '.json'),
             'HIDE_SOURCELINK': False,
diff --git a/nikola/plugins/task/gzip.py b/nikola/plugins/task/gzip.py
--- a/nikola/plugins/task/gzip.py
+++ b/nikola/plugins/task/gzip.py
@@ -28,6 +28,8 @@
 
 import gzip
 import os
+import shlex
+import subprocess
 
 from nikola.plugin_categories import TaskMultiplier
 
@@ -61,13 +63,16 @@ def process(self, task, prefix):
                 gzipped = target + '.gz'
                 gzip_task['file_dep'].append(target)
                 gzip_task['targets'].append(gzipped)
-                gzip_task['actions'].append((create_gzipped_copy, (target, gzipped)))
+                gzip_task['actions'].append((create_gzipped_copy, (target, gzipped, self.site.config['GZIP_COMMAND'])))
         if not flag:
             return []
         return [gzip_task]
 
 
-def create_gzipped_copy(in_path, out_path):
-    with gzip.GzipFile(out_path, 'wb+') as outf:
-        with open(in_path, 'rb') as inf:
-            outf.write(inf.read())
+def create_gzipped_copy(in_path, out_path, command=None):
+    if command:
+        subprocess.check_call(shlex.split(command.format(filename=in_path)))
+    else:
+        with gzip.GzipFile(out_path, 'wb+') as outf:
+            with open(in_path, 'rb') as inf:
+                outf.write(inf.read())


Test patch:
Repository: getnikola/nikola

Issue:
Make nikola work with an empty conf.py
Sane defaults are important. So, let's make Nikola be sane when running with just defaults.



Discussion:

Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -107,6 +107,8 @@ def __init__(self, **config):
             'ADD_THIS_BUTTONS': True,
             'ARCHIVE_PATH': "",
             'ARCHIVE_FILENAME': "archive.html",
+            'BLOG_TITLE': 'Default Title',
+            'BLOG_DESCRIPTION': 'Default Description',
             'BODY_END': "",
             'CACHE_FOLDER': 'cache',
             'CODE_COLOR_SCHEME': 'default',
@@ -173,6 +175,7 @@ def __init__(self, **config):
             'SEARCH_FORM': '',
             'SLUG_TAG_PATH': True,
             'SOCIAL_BUTTONS_CODE': SOCIAL_BUTTONS_CODE,
+            'SITE_URL': 'http://getnikola.com',
             'STORY_INDEX': False,
             'STRIP_INDEXES': False,
             'SITEMAP_INCLUDE_FILELESS_DIRS': True,
diff --git a/nikola/plugins/task/galleries.py b/nikola/plugins/task/galleries.py
--- a/nikola/plugins/task/galleries.py
+++ b/nikola/plugins/task/galleries.py
@@ -368,7 +368,13 @@ def resize_image(self, src, dst, max_size):
                 utils.copy_file(src, dst)
             else:
                 im.save(dst)
+<<<<<<< HEAD
+
+        else:
+            utils.show_msg("WARNING: PIL or Pillow not installed, using original image {0} as thumbnail".format(src))
+=======
         else:  # Image is small
+>>>>>>> c5dc0695e3f216d7d540125aede59a9f49f5bb48
             utils.copy_file(src, dst)
 
     def image_date(self, src):
diff --git a/nikola/plugins/task/listings.py b/nikola/plugins/task/listings.py
--- a/nikola/plugins/task/listings.py
+++ b/nikola/plugins/task/listings.py
@@ -101,7 +101,7 @@ def render_listing(in_name, out_name, folders=[], files=[]):
                 # This is necessary to reflect changes in blog title,
                 # sidebar links, etc.
                 'uptodate': [utils.config_changed(
-                    self.site.config['GLOBAL_CONTEXT'])],
+                    self.site.GLOBAL_CONTEXT)],
                 'clean': True,
             }
             for f in files:
@@ -122,7 +122,7 @@ def render_listing(in_name, out_name, folders=[], files=[]):
                     # This is necessary to reflect changes in blog title,
                     # sidebar links, etc.
                     'uptodate': [utils.config_changed(
-                        self.site.config['GLOBAL_CONTEXT'])],
+                        self.site.GLOBAL_CONTEXT)],
                     'clean': True,
                 }
         if flag:


Test patch:
Repository: getnikola/nikola

Issue:
Add option to sort images in a gallery by name instead of by date,

Add option to sort images in a gallery by name instead of by date,



Discussion:

Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -142,6 +142,7 @@ def __init__(self, **config):
             'FILES_FOLDERS': {'files': ''},
             'FILTERS': {},
             'GALLERY_PATH': 'galleries',
+            'GALLERY_SORT_BY_DATE': True,
             'GZIP_COMMAND': None,
             'GZIP_FILES': False,
             'GZIP_EXTENSIONS': ('.txt', '.htm', '.html', '.css', '.js', '.json'),
diff --git a/nikola/plugins/task/galleries.py b/nikola/plugins/task/galleries.py
--- a/nikola/plugins/task/galleries.py
+++ b/nikola/plugins/task/galleries.py
@@ -66,6 +66,7 @@ def gen_tasks(self):
             'blog_description': self.site.config['BLOG_DESCRIPTION'],
             'use_filename_as_title': self.site.config['USE_FILENAME_AS_TITLE'],
             'gallery_path': self.site.config['GALLERY_PATH'],
+            'sort_by_date': self.site.config['GALLERY_SORT_BY_DATE'],
             'filters': self.site.config['FILTERS'],
             'global_context': self.site.GLOBAL_CONTEXT,
         }
@@ -138,7 +139,10 @@ def gen_tasks(self):
 
             image_list = [x for x in image_list if "thumbnail" not in x]
             # Sort by date
-            image_list.sort(key=lambda a: self.image_date(a))
+            if kw['sort_by_date']:
+                image_list.sort(key=lambda a: self.image_date(a))
+            else:  # Sort by name
+                image_list.sort()
             image_name_list = [os.path.basename(x) for x in image_list]
 
             # List of thumbnail paths


Test patch:
Repository: getnikola/nikola

Issue:
USE_BUNDLES, attribute 'file_dep' must be {list, tuple}
In Py3K map() and filter() return iterators.



Discussion:

Resolving patch:
diff --git a/nikola/plugins/task/bundles.py b/nikola/plugins/task/bundles.py
--- a/nikola/plugins/task/bundles.py
+++ b/nikola/plugins/task/bundles.py
@@ -84,7 +84,7 @@ def build_bundle(output, inputs):
                             for fname in files]
                 file_dep = filter(os.path.isfile, file_dep)  # removes missing files
                 task = {
-                    'file_dep': file_dep,
+                    'file_dep': list(file_dep),
                     'task_dep': ['copy_assets'],
                     'basename': str(self.name),
                     'name': str(output_path),


Test patch:
Repository: getnikola/nikola

Issue:
Tipue task fails on pages/posts with no content.
The Tipue task fails when a page/post has no content and only metadata. The lack of content causes lxml to fail because the document looks empty to it!

```
.  local_search:output/assets/js/tipuesearch_content.json
########################################
TaskError - taskid:local_search:output/assets/js/tipuesearch_content.json
PythonAction Error
Traceback (most recent call last):
  File "/home/devine/Projects/ddevnet-nikola/nikola-env/lib/python2.7/site-packages/doit/action.py", line 343, in execute
    returned_value = self.py_callable(*self.args, **kwargs)
  File "/home/devine/Projects/ddevnet-nikola/nikola-env/lib/python2.7/site-packages/nikola/plugins/task/localsearch/__init__.py", line 77, in save_data
    text = post.text(lang, strip_html=True)
  File "/home/devine/Projects/ddevnet-nikola/nikola-env/lib/python2.7/site-packages/nikola/post.py", line 378, in text
    content = lxml.html.fromstring(data)
  File "/home/devine/Projects/ddevnet-nikola/nikola-env/lib/python2.7/site-packages/lxml/html/__init__.py", line 672, in fromstring
    doc = document_fromstring(html, parser=parser, base_url=base_url, **kw)
  File "/home/devine/Projects/ddevnet-nikola/nikola-env/lib/python2.7/site-packages/lxml/html/__init__.py", line 571, in document_fromstring
    "Document is empty")
ParserError: Document is empty
```

I was initially able to hack around the problem by adding the list of all posts to the list of all pages in `localsearch/__init__.py` by changing:
`posts = self.site.timeline[:]`
to...
`posts = self.site.timeline[:] + self.site.pages[:]`

This works and I don't actually understand why... I came up with this hack while sleep deprived.

 However a better hack is to **put something inside (eg. ".") the post/page file besides the metadata** because if you're defining a page completely within a template whatever you put in that file won't turn up in the template anyway...

Code needs to be added to allow lxml to fail gracefully.

I am on Nikola 6.1.1. I am upgrading my instance from an old one...



Discussion:
So, I saw in IRC that the problem was actually caused by an empty document, so what needs to be done is handle those better.

This looks easy, I'll do a pull-request shortly.


Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -394,8 +394,12 @@ def text(self, lang=None, teaser_only=False, strip_html=False, really_absolute=F
                 data = lxml.html.tostring(document, encoding='unicode')
 
         if data and strip_html:
-            content = lxml.html.fromstring(data)
-            data = content.text_content().strip()  # No whitespace wanted.
+            try:
+                # Not all posts have a body. For example, you may have a page statically defined in the template that does not take content as input.
+                content = lxml.html.fromstring(data)
+                data = content.text_content().strip()  # No whitespace wanted.
+            except lxml.etree.ParserError:
+                data = ""
         return data
 
     @property


Test patch:
Repository: getnikola/nikola

Issue:
Rename post_list options first and last according to python slicing
Rename post_list options first and last according to python slicing[1]:
first -> start, last -> stop. Add some examples to the documentation.

Please assign issue to udono.

[1] http://docs.python.org/2/library/functions.html#slice



Discussion:

Resolving patch:
diff --git a/nikola/plugins/compile/rest/post_list.py b/nikola/plugins/compile/rest/post_list.py
--- a/nikola/plugins/compile/rest/post_list.py
+++ b/nikola/plugins/compile/rest/post_list.py
@@ -30,8 +30,8 @@
 from docutils import nodes
 from docutils.parsers.rst import Directive, directives
 
-from nikola.plugin_categories import RestExtension
 from nikola import utils
+from nikola.plugin_categories import RestExtension
 
 
 class Plugin(RestExtension):
@@ -49,22 +49,25 @@ class PostList(Directive):
     Post List
     =========
     :Directive Arguments: None.
-    :Directive Options: lang, first, last, reverse, tags, template, id
+    :Directive Options: lang, start, stop, reverse, tags, template, id
     :Directive Content: None.
 
     Provides a reStructuredText directive to create a list of posts.
     The posts appearing in the list can be filtered by options.
+    *List slicing* is provided with the *start*, *stop* and *reverse* options.
 
     The following not required options are recognized:
 
-    ``first`` : integer
+    ``start`` : integer
         The index of the first post to show.
+        A negative value like ``-3`` will show the *last* three posts in the
+        post-list.
         Defaults to None.
 
-    ``last`` : integer
+    ``stop`` : integer
         The index of the last post to show.
-        A value of ``-1`` will show every post, but not the last in the
-        post-list.
+        A value negative value like ``-1`` will show every post, but not the
+        *last* in the post-list.
         Defaults to None.
 
     ``reverse`` : flag
@@ -94,11 +97,10 @@ class PostList(Directive):
     ``id`` : string
         A manual id for the post list.
         Defaults to a random name composed by 'post_list_' + uuid.uuid4().hex.
-
     """
     option_spec = {
-        'first': int,
-        'last': int,
+        'start': int,
+        'stop': int,
         'reverse': directives.flag,
         'tags': directives.unchanged,
         'slugs': directives.unchanged,
@@ -109,8 +111,8 @@ class PostList(Directive):
     }
 
     def run(self):
-        first = self.options.get('first')
-        last = self.options.get('last')
+        start = self.options.get('start')
+        stop = self.options.get('stop')
         reverse = self.options.get('reverse', False)
         tags = self.options.get('tags')
         tags = [t.strip().lower() for t in tags.split(',')] if tags else []
@@ -128,7 +130,7 @@ def run(self):
         else:
             timeline = [p for p in self.site.timeline if p.use_in_feeds]
 
-        for post in timeline[first:last:step]:
+        for post in timeline[start:stop:step]:
             if tags:
                 cont = True
                 for tag in tags:


Test patch:
Repository: getnikola/nikola

Issue:
Requires nikola commands to be run in root only
Contrary to e.g. git, nikola does not recognize its installation subdir as "an existing Nikola site".

Steps to reproduce:

```
$ nikola new_post
Creating New Post
-----------------

Enter title: hello
Scanning posts....done!
[2013-11-15T20:16:18Z] NOTICE: new_post: Your post's text is at: posts/hello.rst
$ cd posts/
$ nikola build
[2013-11-15T20:16:50Z] ERROR: Nikola: This command needs to run inside an existing Nikola site.
```



Discussion:

Resolving patch:
diff --git a/nikola/main.py b/nikola/main.py
--- a/nikola/main.py
+++ b/nikola/main.py
@@ -42,7 +42,7 @@
 
 from . import __version__
 from .nikola import Nikola
-from .utils import _reload, sys_decode, LOGGER, STRICT_HANDLER
+from .utils import _reload, sys_decode, get_root_dir, LOGGER, STRICT_HANDLER
 
 
 config = {}
@@ -58,6 +58,11 @@ def main(args):
         nullhandler.push_application()
         quiet = True
     global config
+
+    root = get_root_dir()
+    if root:
+        os.chdir(root)
+
     sys.path.append('')
     try:
         import conf
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -807,3 +807,20 @@ def demote_headers(doc, level=1):
             elements = doc.xpath('//h' + str(i))
             for e in elements:
                 e.tag = 'h' + str(i + level)
+
+
+def get_root_dir():
+    """Find root directory of nikola installation by looking for conf.py"""
+    root = os.getcwd()
+
+    while True:
+        if os.path.exists(os.path.join(root, 'conf.py')):
+            return root
+        else:
+            basedir = os.path.split(root)[0]
+            # Top directory, already checked
+            if basedir == root:
+                break
+            root = basedir
+
+    return None


Test patch:
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -338,5 +338,16 @@ def test_monthly_archive(self):
         self.assertTrue(os.path.isfile(os.path.join(self.tmpdir, 'target', 'output', '2012', '03', 'index.html')))
 
 
+class SubdirRunningTest(DemoBuildTest):
+    """Check that running nikola from subdir works."""
+
+    def test_subdir_run(self):
+        """Check whether build works from posts/"""
+
+        with cd(os.path.join(self.target_dir, 'posts')):
+            result = main.main(['build'])
+            self.assertEquals(result, 0)
+
+
 if __name__ == "__main__":
     unittest.main()
Repository: getnikola/nikola

Issue:
Thumbnail generation fails with no reason
At the moment:

```
[2013-11-19T12:48:09Z] WARNING: render_galleries: \
Can't thumbnail galleries/xbox/ori-label.jpg, using original image as thumbnail
```

Should be:

```
[2013-11-19T12:48:09Z] WARNING: render_galleries: \
Can't thumbnail galleries/xbox/ori-label.jpg, using original image as thumbnail
[2013-11-19T12:48:09Z] WARNING: render_galleries: \
decoder jpeg not available
```

Or something like this



Discussion:
Well... PIL and Pillow are rather uncooperative about the reasons for failure :-/


Resolving patch:
diff --git a/nikola/plugins/task/galleries.py b/nikola/plugins/task/galleries.py
--- a/nikola/plugins/task/galleries.py
+++ b/nikola/plugins/task/galleries.py
@@ -540,8 +540,9 @@ def resize_image(self, src, dst, max_size):
             try:
                 im.thumbnail(size, Image.ANTIALIAS)
                 im.save(dst)
-            except Exception:
-                self.logger.warn("Can't thumbnail {0}, using original image as thumbnail".format(src))
+            except Exception, e:
+                self.logger.warn("Can't thumbnail {0}, using original "
+                                 "image as thumbnail ({1})".format(src, e))
                 utils.copy_file(src, dst)
         else:  # Image is small
             utils.copy_file(src, dst)


Test patch:
Repository: getnikola/nikola

Issue:
Make the template for tag index pages configurable.
Tag index pages currently use the same template as other index pages.
Make this configurable.



Discussion:
Instead of yet another config value, I suggest the following (includes backwards compatibility):
1. have a `tag.tmpl` file in our themes, which just `include`s the old `index.tmpl`
2. set template_name to `tag.tmpl`

I’m 99% sure it won’t break anything, because it would just take it from base and base-jinja (the users of which would need to update their theme).  But in case it would, we could always employ a trick of **if** `tag.tmpl` exists → use it; **else** → use `index.tmpl`; **fi**

I like @kwpolska's suggestion. Also, it will need documenting in theming.txt either way

Implementing on branch `tag-index-page-2`.


Resolving patch:
diff --git a/nikola/plugins/task/tags.py b/nikola/plugins/task/tags.py
--- a/nikola/plugins/task/tags.py
+++ b/nikola/plugins/task/tags.py
@@ -189,7 +189,7 @@ def page_name(tagname, i, lang):
             return name
 
         # FIXME: deduplicate this with render_indexes
-        template_name = "index.tmpl"
+        template_name = "tagindex.tmpl"
         # Split in smaller lists
         lists = []
         while post_list:


Test patch:
Repository: getnikola/nikola

Issue:
nikola auto X doit auto
The nikola auto commands hides the doit auto command.

Both are useful...
What about renaming nikola auto to "liveroload" this way we can use both?

if you guys agree I will make a pull request.



Discussion:
`livereload*`.  And it looks like `doit auto` works just like `nikola auto` but without the livereload/serve part.  Although it looks like it is done differently than we do it, and it does not support Windows.  Maybe create a `doit_auto` alias so the more likely and more workable use-case of `livereload` is the default?


Resolving patch:
diff --git a/nikola/main.py b/nikola/main.py
--- a/nikola/main.py
+++ b/nikola/main.py
@@ -38,6 +38,7 @@
 from doit.cmd_help import Help as DoitHelp
 from doit.cmd_run import Run as DoitRun
 from doit.cmd_clean import Clean as DoitClean
+from doit.cmd_auto import Auto as DoitAuto
 from logbook import NullHandler
 
 from . import __version__
@@ -132,6 +133,10 @@ def clean_tasks(self, tasks, dryrun):
                 shutil.rmtree(cache_folder)
         return super(Clean, self).clean_tasks(tasks, dryrun)
 
+# Nikola has its own "auto" commands that uses livereload.
+# Expose original doit "auto" command as "doit_auto".
+DoitAuto.name = 'doit_auto'
+
 
 class NikolaTaskLoader(TaskLoader):
     """custom task loader to get tasks from Nikola instead of dodo.py file"""
@@ -161,7 +166,7 @@ def load_tasks(self, cmd, opt_values, pos_args):
 
 class DoitNikola(DoitMain):
     # overwite help command
-    DOIT_CMDS = list(DoitMain.DOIT_CMDS) + [Help, Build, Clean]
+    DOIT_CMDS = list(DoitMain.DOIT_CMDS) + [Help, Build, Clean, DoitAuto]
     TASK_LOADER = NikolaTaskLoader
 
     def __init__(self, nikola, quiet=False):


Test patch:
Repository: getnikola/nikola

Issue:
`nikola -V/--version` doesn’t work outside of a site
```
(nikola-py2)[kwpolska@kw-cassandra ~WORKON_HOME]% nikola --version
[2014-01-04T17:40:42Z] ERROR: Nikola: This command needs to run inside an existing Nikola site.
(nikola-py2)[kwpolska@kw-cassandra ~WORKON_HOME]% cd ~/nikola
(nikola-py2)[kwpolska@kw-cassandra nikola Gmaster]% nikola --version
Nikola version 6.2.1
```



Discussion:

Resolving patch:
diff --git a/nikola/main.py b/nikola/main.py
--- a/nikola/main.py
+++ b/nikola/main.py
@@ -191,7 +191,9 @@ def run(self, cmd_args):
             args = ['help']
             # Hide run because Nikola uses build
             sub_cmds.pop('run')
-
+        if len(args) == 0 or any(arg in ["--version", '-V'] for arg in args):
+            cmd_args = ['version']
+            args = ['version']
         if len(args) == 0 or args[0] not in sub_cmds.keys() or \
                 args[0] == 'build':
             # Check for conf.py before launching run
@@ -203,4 +205,4 @@ def run(self, cmd_args):
 
     @staticmethod
     def print_version():
-        print("Nikola version " + __version__)
+        print("Nikola v" + __version__)
diff --git a/nikola/plugins/command/version.py b/nikola/plugins/command/version.py
--- a/nikola/plugins/command/version.py
+++ b/nikola/plugins/command/version.py
@@ -41,4 +41,4 @@ class CommandVersion(Command):
 
     def _execute(self, options={}, args=None):
         """Print the version number."""
-        print("Nikola version " + __version__)
+        print("Nikola v" + __version__)


Test patch:
Repository: getnikola/nikola

Issue:
HIDE_SOURCELINK configuration seems to be buggy
Nikola 6.3.0, default configuration, source links are shown:

```
# Hide link to source for the posts?
# HIDE_SOURCELINK = False
# Copy the source files for your pages?
# Setting it to False implies HIDE_SOURCELINK = True
# COPY_SOURCES = True
```

The same configuration (default) with comments removed:

```
# Hide link to source for the posts?
HIDE_SOURCELINK = False
# Copy the source files for your pages?
# Setting it to False implies HIDE_SOURCELINK = True
COPY_SOURCES = True
```

In this case source links are hidden and for me it looks like a bug.



Discussion:
Confirmed. Working on it.


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -305,7 +305,7 @@ def __init__(self, **config):
         if config.get('PRETTY_URLS', False) and 'STRIP_INDEXES' not in config:
             self.config['STRIP_INDEXES'] = True
 
-        if config.get('COPY_SOURCES') and not self.config['HIDE_SOURCELINK']:
+        if not config.get('COPY_SOURCES'):
             self.config['HIDE_SOURCELINK'] = True
 
         self.config['TRANSLATIONS'] = self.config.get('TRANSLATIONS',


Test patch: