internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getnikola/nikola

Issue:
Sites with large image galleries are always slow to build
Currently in gallery.py, there is this code:

```
            if self.kw['sort_by_date']:
                image_list.sort(key=lambda a: self.image_date(a))
```

Sadly this means we open and read the EXIF data of every image every build, even if there's nothing to be done.

In a site with 1400  images and nothing to rebuild:

```
$ time nikola build 
Scanning posts....done!

real    0m3.165s
user    0m3.008s
sys     0m0.152s
```

And the same thing without sorting by date:

```
$ time nikola build 
Scanning posts....done!

real    0m2.184s
user    0m2.028s
sys     0m0.140s
```

This is a device with a very fast SSD, so other users may feel it even more. Surely that EXIF read can be avoided.



Discussion:

Resolving patch:
diff --git a/nikola/plugins/task/galleries.py b/nikola/plugins/task/galleries.py
--- a/nikola/plugins/task/galleries.py
+++ b/nikola/plugins/task/galleries.py
@@ -194,13 +194,6 @@ def gen_tasks(self):
             # Create image list, filter exclusions
             image_list = self.get_image_list(gallery)
 
-            # Sort as needed
-            # Sort by date
-            if self.kw['sort_by_date']:
-                image_list.sort(key=lambda a: self.image_date(a))
-            else:  # Sort by name
-                image_list.sort()
-
             # Create thumbnails and large images in destination
             for image in image_list:
                 for task in self.create_target_images(image, input_folder):
@@ -558,6 +551,18 @@ def url_from_path(p):
             url = '/'.join(os.path.relpath(p, os.path.dirname(output_name) + os.sep).split(os.sep))
             return url
 
+        all_data = list(zip(img_list, thumbs, img_titles))
+
+        if self.kw['sort_by_date']:
+            all_data.sort(key=lambda a: self.image_date(a[0]))
+        else:  # Sort by name
+            all_data.sort(key=lambda a: a[0])
+
+        if all_data:
+            img_list, thumbs, img_titles = zip(*all_data)
+        else:
+            img_list, thumbs, img_titles = [], [], []
+
         photo_array = []
         for img, thumb, title in zip(img_list, thumbs, img_titles):
             w, h = _image_size_cache.get(thumb, (None, None))
@@ -591,6 +596,18 @@ def gallery_rss(self, img_list, dest_img_list, img_titles, lang, permalink, outp
         def make_url(url):
             return urljoin(self.site.config['BASE_URL'], url.lstrip('/'))
 
+        all_data = list(zip(img_list, dest_img_list, img_titles))
+
+        if self.kw['sort_by_date']:
+            all_data.sort(key=lambda a: self.image_date(a[0]))
+        else:  # Sort by name
+            all_data.sort(key=lambda a: a[0])
+
+        if all_data:
+            img_list, dest_img_list, img_titles = zip(*all_data)
+        else:
+            img_list, dest_img_list, img_titles = [], [], []
+
         items = []
         for img, srcimg, title in list(zip(dest_img_list, img_list, img_titles))[:self.kw["feed_length"]]:
             img_size = os.stat(


Test patch:
Repository: getnikola/nikola

Issue:
NIkola does not see plugins listed under DISABLED_PLUGINS
First noticed while running `nikola build`:
`ERROR: _switch to py3:please!. Task dependency 'sitemap' does not exist.`

> polyzen> there's no longer a way to disable the sitemap? i don't see it under `nikola plugin --list-installed` nor `nikola list` unless it's part of render_site
> +ralsina> sitemap at /home/ralsina/Desktop/proyectos/nikola/master/nikola/plugins/task/sitemap/
> +ralsina> I suspect disabled plugins are not listed anymore, try that in some other folder
> polyzen> ah yes



Discussion:
There are two different problems here:
1. `DISABLED_PLUGINS`:
   
   a. Listing plugins bases off plugins that are enabled → hard to fix, and we probably won’t do it
   b. Listing tasks bases off plugins that are enabled → won’t fix, because it would require a huge architecture change
2. You disabled the sitemaps plugin, and our py2 warning depends on it.

After removing robots and sitemap from disabled plugins, I no longer get an error like this.
`DISABLED_PLUGINS = ["render_archive","render_galleries","render_indexes","render_listings","render_tags"]`

The error appeared because

> You disabled the sitemaps plugin, and our py2 warning depends on [the sitemap task so the warning is shown in nikola build as the last thing].

@Kwpolska Yes, but that's not what this issue is about :-)

Before I worked on loading only the needed plugins, they would all appear on --list-installed and now they don't.


Resolving patch:
diff --git a/nikola/plugins/command/plugin.py b/nikola/plugins/command/plugin.py
--- a/nikola/plugins/command/plugin.py
+++ b/nikola/plugins/command/plugin.py
@@ -176,8 +176,11 @@ def list_installed(self):
             plugins.append([plugin.name, p])
 
         plugins.sort()
+        print('Installed Plugins')
+        print('-----------------')
         for name, path in plugins:
             print('{0} at {1}'.format(name, path))
+        print('\n\nAlso, you have disabled these plugins: {}'.format(self.site.config['DISABLED_PLUGINS']))
         return 0
 
     def do_upgrade(self, url):


Test patch:
Repository: getnikola/nikola

Issue:
anchor links are improperly rewritten
When writing raw html pages that contain local anchor links they get rewritten improperly. Example

Create a page with the following content

.. raw:: html

```
<a href="#foo">a link</a>
```

When building that page, the generated html will look like

<a href=".#foo">a link</a>

but should really look like

<a href="#foo">a link</a>



Discussion:
On default configuration this works correctly, so it looks like interaction with "something" in the config. Following up.

Even though it does something that works, url_replacer sure is failing to fullfill the "\* Leaves fragments unchanged" part of its docstring.

I followed these steps

virtualenv env
source env/bin/activate
pip install nikola
pip install "Pillow<3.0" (to avoid a bug with Pillow 3.0)

nikola init --demo foo
nikola new_page

created the page by adding

```
.. raw:: html

    <a href="#foo">a link</a>
```

nikola build
nikola serve

checked out the html in the browser

see this

```
<a href=".#foo">a link</a>
```

What does this page’s permalink look like? Are you sure the source is rewritten in the files? or is it just the browser’s interpretation (could be `<base>` related)?

Ok, with help of pdb, I found where the anchor is being incorrectly rewritten.

Inside url_replacer (in nikola.py) around line 1241 you find

``` python
if not result:
    result = "."
```

This is the culprit of the bad rewrite. From here on, the fragment is properly added to result later on, but the original anchor "#foo" is rewritten as ".#foo".

Haven't spent enough time following the code to come up with a proper patch for this, but in time I might.

A very ugly way of avoiding the issue would be to replace that conditional with

``` python
if not result and not parsed_dst.fragment:
    result = "."
```

but this would only address this particular case so I doubt it's the proper fix

Actually that seems to me to be the right fix.


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -1277,7 +1277,7 @@ def url_replacer(self, src, dst, lang=None, url_type=None):
         # Now i is the longest common prefix
         result = '/'.join(['..'] * (len(src_elems) - i - 1) + dst_elems[i:])
 
-        if not result:
+        if not result and not parsed_dst.fragment:
             result = "."
 
         # Don't forget the query part of the link


Test patch:
Repository: getnikola/nikola

Issue:
 The `IPython.nbconvert` package has been deprecated.
IPython 4.0 was just released (http://blog.jupyter.org/2015/08/12/first-release-of-jupyter/).  The `IPython.nbconvert` package has been deprecated. The [`nbconvert`-based ipynb compiler](https://github.com/getnikola/nikola/blob/15217bc93f0af0d70ffe33e0ea067d81ddf32403/nikola/plugins/compile/ipynb.py) should be updated where necessary to reflect any API changes in the 4.0 release.



Discussion:
Yep... for now the shims in Jupyter should work OK, but we need to make the update. Thanks for opening the issue!!


Resolving patch:
diff --git a/nikola/plugins/compile/ipynb.py b/nikola/plugins/compile/ipynb.py
--- a/nikola/plugins/compile/ipynb.py
+++ b/nikola/plugins/compile/ipynb.py
@@ -32,21 +32,33 @@
 import sys
 
 try:
-    import IPython
-    from IPython.nbconvert.exporters import HTMLExporter
-    if IPython.version_info[0] >= 3:     # API changed with 3.0.0
-        from IPython import nbformat
-        current_nbformat = nbformat.current_nbformat
-        from IPython.kernel import kernelspec
-    else:
-        import IPython.nbformat.current as nbformat
-        current_nbformat = 'json'
-        kernelspec = None
-
-    from IPython.config import Config
+    from nbconvert.exporters import HTMLExporter
+    import nbformat
+    current_nbformat = nbformat.current_nbformat
+    from jupyter_client import kernelspec
+    from traitlets.config import Config
     flag = True
+    ipy_modern = True
 except ImportError:
-    flag = None
+    try:
+        import IPython
+        from IPython.nbconvert.exporters import HTMLExporter
+        if IPython.version_info[0] >= 3:     # API changed with 3.0.0
+            from IPython import nbformat
+            current_nbformat = nbformat.current_nbformat
+            from IPython.kernel import kernelspec
+            ipy_modern = True
+        else:
+            import IPython.nbformat.current as nbformat
+            current_nbformat = 'json'
+            kernelspec = None
+            ipy_modern = False
+
+        from IPython.config import Config
+        flag = True
+    except ImportError:
+        flag = None
+        ipy_modern = None
 
 from nikola.plugin_categories import PageCompiler
 from nikola.utils import makedirs, req_missing, get_logger, STDERR_HANDLER
@@ -118,7 +130,7 @@ def create_post(self, path, **kw):
             # imported .ipynb file, guaranteed to start with "{" because it’s JSON.
             nb = nbformat.reads(content, current_nbformat)
         else:
-            if IPython.version_info[0] >= 3:
+            if ipy_modern:
                 nb = nbformat.v4.new_notebook()
                 nb["cells"] = [nbformat.v4.new_markdown_cell(content)]
             else:
@@ -151,7 +163,7 @@ def create_post(self, path, **kw):
             nb["metadata"]["nikola"] = metadata
 
         with io.open(path, "w+", encoding="utf8") as fd:
-            if IPython.version_info[0] >= 3:
+            if ipy_modern:
                 nbformat.write(nb, fd, 4)
             else:
                 nbformat.write(nb, fd, 'ipynb')


Test patch:
Repository: getnikola/nikola

Issue:
Use a FS cache for Jinja to speed it up a bit
http://jinja.pocoo.org/docs/dev/api/#bytecode-cache



Discussion:

Resolving patch:
diff --git a/nikola/plugins/template/jinja.py b/nikola/plugins/template/jinja.py
--- a/nikola/plugins/template/jinja.py
+++ b/nikola/plugins/template/jinja.py
@@ -52,18 +52,21 @@ def __init__(self):
         """Initialize Jinja2 environment with extended set of filters."""
         if jinja2 is None:
             return
-        self.lookup = jinja2.Environment()
+
+    def set_directories(self, directories, cache_folder):
+        """Create a new template lookup with set directories."""
+        if jinja2 is None:
+            req_missing(['jinja2'], 'use this theme')
+        cache_folder = os.path.join(cache_folder, 'jinja')
+        makedirs(cache_folder)
+        cache = jinja2.FileSystemBytecodeCache(cache_folder)
+        self.lookup = jinja2.Environment(bytecode_cache=cache)
         self.lookup.trim_blocks = True
         self.lookup.lstrip_blocks = True
         self.lookup.filters['tojson'] = json.dumps
         self.lookup.globals['enumerate'] = enumerate
         self.lookup.globals['isinstance'] = isinstance
         self.lookup.globals['tuple'] = tuple
-
-    def set_directories(self, directories, cache_folder):
-        """Create a new template lookup with set directories."""
-        if jinja2 is None:
-            req_missing(['jinja2'], 'use this theme')
         self.directories = directories
         self.create_lookup()
 


Test patch:
Repository: getnikola/nikola

Issue:
MathJax isn't enabled for posts in non-default language only
If there's a math post only in non-default language, Nikola won't enable MathJax there
1) Create new site: `nikola init non-def-math`. All the default values will suffice, just set at least two languages:

```
Language(s) to use [en]: en,pl
```

2) Create a Polish-only post:

```
$ nikola new_post -t "Math test" --tags=mathjax
$ echo '\nEuler’s formula: :math:`e^{ix} = \\cos x + i\\sin x`' >> posts/math-test.rst
$ mv posts/math-test.rst posts/math-test.pl.rst
```

3) `nikola build` will build a post without MathJax:

```
 $ nikola build && grep -r cdn.mathjax.org output || echo "Nope" 
Nope
```

If post has versions for both languages, maths are enabled correctly on both pages.

I did dig a bit in source code, and it seems like [mathjax is tested for presence in tags only for default language](https://github.com/getnikola/nikola/commit/e068305a4eb2099adca3b53e13c35c01b979c367/nikola/post.py#L283) (`nikola.utils.LocaleBorg().current_lang` returns default language).



Discussion:
Good catch!


Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -109,7 +109,6 @@ def __init__(
         self.base_url = self.config['BASE_URL']
         self.is_draft = False
         self.is_private = False
-        self.is_mathjax = False
         self.strip_indexes = self.config['STRIP_INDEXES']
         self.index_file = self.config['INDEX_FILE']
         self.pretty_urls = self.config['PRETTY_URLS']
@@ -223,9 +222,6 @@ def __init__(
         self.use_in_feeds = use_in_feeds and not is_draft and not is_private \
             and not self.publish_later
 
-        # If mathjax is a tag, or it's a ipynb post, then enable mathjax rendering support
-        self.is_mathjax = ('mathjax' in self.tags) or (self.compiler.name == 'ipynb')
-
         # Register potential extra dependencies
         self.compiler.register_extra_dependencies(self)
 
@@ -258,6 +254,17 @@ def _has_pretty_url(self, lang):
         else:
             return False
 
+    @property
+    def is_mathjax(self):
+        """True if this post has the mathjax tag in the current language or is a python notebook."""
+        if self.compiler.name == 'ipynb':
+            return True
+        lang = nikola.utils.LocaleBorg().current_lang
+        if self.is_translation_available(lang):
+            return 'mathjax' in self.tags_for_language(lang)
+        # If it has math in ANY other language, enable it. Better inefficient than broken.
+        return 'mathjax' in self.alltags
+
     @property
     def alltags(self):
         """Return ALL the tags for this post."""


Test patch:
Repository: getnikola/nikola

Issue:
deploy crashes with state system
Will investigate later.

``` pytb
Traceback (most recent call last):
  File "/home/kwpolska/virtualenvs/nikola-py3/lib/python3.5/site-packages/doit/doit_cmd.py", line 168, in run
    return command.parse_execute(args)
  File "/home/kwpolska/virtualenvs/nikola-py3/lib/python3.5/site-packages/doit/cmd_base.py", line 122, in parse_execute
    return self.execute(params, args)
  File "/home/kwpolska/git/nikola/nikola/plugin_categories.py", line 124, in execute
    return self._execute(options, args)
  File "/home/kwpolska/git/nikola/nikola/plugins/command/deploy.py", line 135, in _execute
    self.site.state.set('last_deploy', new_deploy.isoformat())
  File "/home/kwpolska/git/nikola/nikola/state.py", line 64, in set
    self._save()
  File "/home/kwpolska/git/nikola/nikola/state.py", line 82, in _save
    json.dump(self._local.data, outf, sort_keys=True, indent=2)
  File "/usr/lib64/python3.5/json/__init__.py", line 179, in dump
    fp.write(chunk)
  File "/home/kwpolska/virtualenvs/nikola-py3/lib/python3.5/tempfile.py", line 483, in func_wrapper
    return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'
```



Discussion:

Resolving patch:
diff --git a/nikola/state.py b/nikola/state.py
--- a/nikola/state.py
+++ b/nikola/state.py
@@ -78,6 +78,11 @@ def _read(self):
     def _save(self):
         dname = os.path.dirname(self._path)
         with tempfile.NamedTemporaryFile(dir=dname, delete=False) as outf:
+            # TODO replace with encoding='utf-8' and mode 'w+' in v8
             tname = outf.name
-            json.dump(self._local.data, outf, sort_keys=True, indent=2)
+            data = json.dumps(self._local.data, sort_keys=True, indent=2)
+            try:
+                outf.write(data)
+            except TypeError:
+                outf.write(data.encode('utf-8'))
         shutil.move(tname, self._path)


Test patch:
Repository: getnikola/nikola

Issue:
jinjify: convert a single mako template file to jinja?
It looks like jinjify requires a full theme to work. For example, how do you synchronize the two templates in the [projectpages plugin](https://github.com/getnikola/plugins/tree/master/v7/projectpages/templates) with jinjify?



Discussion:
Patches welcome. I did the conversions there by creating fake “themes” with templates/ directories and pointing jinjify at them.


Resolving patch:
diff --git a/scripts/jinjify.py b/scripts/jinjify.py
--- a/scripts/jinjify.py
+++ b/scripts/jinjify.py
@@ -6,6 +6,7 @@
 import re
 import json
 import shutil
+import tempfile
 
 import colorama
 import jinja2
@@ -229,6 +230,11 @@ def jinjify_shortcodes(in_dir, out_dir):
             outf.write(data)
             
 
+def usage():
+    print("Usage: python {} [in-dir] [out-dir]".format(sys.argv[0]))
+    print("OR")
+    print("Usage: python {} [in-file] [out-file]".format(sys.argv[0]))
+
 if __name__ == "__main__":
     if len(sys.argv) == 1:
         print('Performing standard conversions:')
@@ -240,6 +246,18 @@ def jinjify_shortcodes(in_dir, out_dir):
             jinjify(m, j)
         jinjify_shortcodes('nikola/data/shortcodes/mako', 'nikola/data/shortcodes/jinja')
     elif len(sys.argv) != 3:
-        print('ERROR: needs input and output directory, or no arguments for default conversions.')
-    else:
+        print('ERROR: needs input and output directory (file), or no arguments for default conversions.')
+        usage()
+    elif os.path.isdir(sys.argv[1]) and (os.path.isdir(sys.argv[2]) or not os.path.exists(sys.argv[2])):
         jinjify(sys.argv[1], sys.argv[2])
+    elif os.path.isfile(sys.argv[1]) and (os.path.isfile(sys.argv[2]) or not os.path.exists(sys.argv[2])):
+        tmpdir = tempfile.mkdtemp()
+        indir = os.path.sep.join((tmpdir, 'in', 'templates'))
+        outdir = os.path.sep.join((tmpdir, 'out', 'templates'))
+        os.makedirs(indir)
+        shutil.copy(sys.argv[1], indir)
+        jinjify(os.path.dirname(indir), os.path.dirname(outdir))
+        shutil.move(os.path.sep.join((outdir, os.path.basename(sys.argv[1]))), sys.argv[2])
+    else:
+        print('ERROR: the two arguments must be both directories or files')
+        usage()


Test patch:
Repository: getnikola/nikola

Issue:
Untranslated posts appear in sitemap.xml
Recently upgraded to 7.7.6 and received a warning from Google Webmaster Tools about increased number of 404s. It looks like sitemap.xml is the cause:

```
$ nikola --version
Nikola vv7.7.6
```

conf.py:

```
# What is the default language?
DEFAULT_LANG = "en"

# What other languages do you have?
# The format is {"translationcode" : "path/to/translation" }
# the path will be used as a prefix for the generated pages location
TRANSLATIONS = {
    DEFAULT_LANG: "",
    "ru": "./ru",
}

SHOW_UNTRANSLATED_POSTS = False
```

```
% tree stories
stories
└── example-post.md
```

sitemap.xml:

```
 <url>
  <loc>https://example.com/stories/example-post/</loc>
  <lastmod>2016-03-07T13:07:00Z</lastmod>
  <xhtml:link rel="alternate" hreflang="ru" href="https://example.com/ru/stories/example-post/" />
 </url>
```



Discussion:
Ack, I will take a look.


Resolving patch:
diff --git a/nikola/plugins/task/sitemap/__init__.py b/nikola/plugins/task/sitemap/__init__.py
--- a/nikola/plugins/task/sitemap/__init__.py
+++ b/nikola/plugins/task/sitemap/__init__.py
@@ -158,7 +158,7 @@ def scan_locs():
                         continue
                     alternates = []
                     if post:
-                        for lang in kw['translations']:
+                        for lang in post.translated_to:
                             alt_url = post.permalink(lang=lang, absolute=True)
                             if encodelink(loc) == alt_url:
                                 continue
@@ -215,7 +215,7 @@ def scan_locs():
                         loc = urljoin(base_url, base_path + path)
                         alternates = []
                         if post:
-                            for lang in kw['translations']:
+                            for lang in post.translated_to:
                                 alt_url = post.permalink(lang=lang, absolute=True)
                                 if encodelink(loc) == alt_url:
                                     continue


Test patch:
Repository: getnikola/nikola

Issue:
Give full 'auto' site URL [idea]
When starting the auto mode, here's what one can see : 

```
C:\PORT-STC\PRIVATE\pilgrimp>nikola auto
Scanning posts......done!
[2016-05-03T06:17:36Z] INFO: auto: Watching files for changes...
[2016-05-03T06:17:36Z] INFO: ws4py: Using select as epoll is not available
[2016-05-03T06:17:36Z] INFO: auto: Serving HTTP on 127.0.0.1 port 8000...
```

The "problem" is that if I try to open "http://127.0.0.1:8000" in my browser, I get an error. What I need to open is actually : "http://127.0.0.1:8000/index.html".  Somehow, it took me five minutes to figure it out :(

So, I propose this in the log : 

```
C:\PORT-STC\PRIVATE\pilgrimp>nikola auto
Scanning posts......done!
[2016-05-03T06:17:36Z] INFO: auto: Watching files for changes...
[2016-05-03T06:17:36Z] INFO: ws4py: Using select as epoll is not available
[2016-05-03T06:17:36Z] INFO: auto: Serving HTTP on 127.0.0.1 port 8000...
[2016-05-03T06:17:36Z] INFO You can browse your site at http://127.0.0.1:8000/index.html
```

Best regards,

Stefan



Discussion:
It looks like a Windows quirk.


Resolving patch:
diff --git a/nikola/plugins/command/auto/__init__.py b/nikola/plugins/command/auto/__init__.py
--- a/nikola/plugins/command/auto/__init__.py
+++ b/nikola/plugins/command/auto/__init__.py
@@ -302,8 +302,8 @@ def serve_static(self, environ, start_response):
         mimetype = 'text/html' if uri.endswith('/') else mimetypes.guess_type(uri)[0] or 'application/octet-stream'
 
         if os.path.isdir(f_path):
-            if not f_path.endswith('/'):  # Redirect to avoid breakage
-                start_response('301 Redirect', [('Location', p_uri.path + '/')])
+            if not p_uri.path.endswith('/'):  # Redirect to avoid breakage
+                start_response('301 Moved Permanently', [('Location', p_uri.path + '/')])
                 return []
             f_path = os.path.join(f_path, self.site.config['INDEX_FILE'])
             mimetype = 'text/html'


Test patch: