internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getnikola/nikola

Issue:
html5.js 404 for IE
Got several 404s for html5.js in access.log. The following snippet

```
    %if use_cdn:
        <!--[if lt IE 9]><script src="//html5shim.googlecode.com/svn/trunk/html5.js"></script><![endif]-->
    %else:
        <!--[if lt IE 9]><script src="/assets/js/html5.js"></script><![endif]-->
    %endif
```

is expanded as

```
        <!--[if lt IE 9]><script src="/assets/js/html5.js"></script><![endif]-->
```

with `BASE_URL = "http://example.com/blog/"` and `USE_CDN = False`. Expected path should be `http://example.com/blog/assets/js/html5.js`.



Discussion:
Out of interest, why aren’t coments processed?

Because why would they be?  Usually, HTML comments are supposed to be left in as-is, someone might expect to see an exact thing there.  Or, they should be removed.  But changing it makes no sense.  lxml does not see this as a `<script>` tag, just a comment containing something that it should not care about in the slightest.

Maybe explicit jinja context variable is the way to fix this?

```
 <!--[if lt IE 9]><script src="{{ BLOG_PREFIX }}/assets/js/html5.js"></script><![endif]-->
```

We do not have a BLOG_PREFIX variable, we have a link replacement facility though.

https://github.com/getnikola/nikola/blob/master/nikola/nikola.py#L799

With some work, we can make it use that.


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -650,6 +650,7 @@ def __init__(self, **config):
             'SHOW_SOURCELINK')
         self._GLOBAL_CONTEXT['extra_head_data'] = self.config.get('EXTRA_HEAD_DATA')
         self._GLOBAL_CONTEXT['colorbox_locales'] = LEGAL_VALUES['COLORBOX_LOCALES']
+        self._GLOBAL_CONTEXT['url_replacer'] = self.url_replacer
 
         self._GLOBAL_CONTEXT.update(self.config.get('GLOBAL_CONTEXT', {}))
 


Test patch:
Repository: getnikola/nikola

Issue:
Post metadata changes don’t guarantee rebuilds
If you change post metadata, and only post metadata, a rebuild of indexes and the post page might not happen.  `title` and `slug` are immune to this; `tags` rebuilds tag pages, but nothing else; and any other metadata don’t do a thing.



Discussion:
When you say title and slug are immune, do you mean that changes to those fields always trigger a rebuild or they do not trigger a rebuild?

Always trigger.


Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -318,8 +318,10 @@ def deps(self, lang):
         deps = []
         if self.default_lang in self.translated_to:
             deps.append(self.base_path)
+            deps.append(self.source_path)
         if lang != self.default_lang:
-            deps += [get_translation_candidate(self.config, self.base_path, lang)]
+            deps.append(get_translation_candidate(self.config, self.base_path, lang))
+            deps.append(get_translation_candidate(self.config, self.source_path, lang))
         return deps
 
     def compile(self, lang):


Test patch:
Repository: getnikola/nikola

Issue:
SHOW_UNTRANSLATED_POSTS=False leads to 404s in archives
Even though SHOW_UNTRANSLATED_POSTS=False, archives still display posts that do not exist, leading to 404s everywhere.



Discussion:

Resolving patch:
diff --git a/nikola/plugins/task/archive.py b/nikola/plugins/task/archive.py
--- a/nikola/plugins/task/archive.py
+++ b/nikola/plugins/task/archive.py
@@ -90,6 +90,7 @@ def gen_tasks(self):
             "filters": self.site.config['FILTERS'],
             "create_monthly_archive": self.site.config['CREATE_MONTHLY_ARCHIVE'],
             "create_single_archive": self.site.config['CREATE_SINGLE_ARCHIVE'],
+            "show_untranslated_posts": self.site.config['SHOW_UNTRANSLATED_POSTS'],
             "create_full_archives": self.site.config['CREATE_FULL_ARCHIVES'],
             "create_daily_archive": self.site.config['CREATE_DAILY_ARCHIVE'],
         }
@@ -109,6 +110,11 @@ def gen_tasks(self):
                 # if we are creating one single archive, or full archives
                 archdata[None] = self.site.posts  # for create_single_archive
 
+            # Filter untranslated posts (Issue #1360)
+            if not kw["show_untranslated_posts"]:
+                for year, posts in archdata.items():
+                    archdata[year] = [p for p in posts if lang in p.translated_to]
+
             for year, posts in archdata.items():
                 # Add archive per year or total archive
                 if year:


Test patch:
Repository: getnikola/nikola

Issue:
Post metadata changes don’t guarantee rebuilds
If you change post metadata, and only post metadata, a rebuild of indexes and the post page might not happen.  `title` and `slug` are immune to this; `tags` rebuilds tag pages, but nothing else; and any other metadata don’t do a thing.



Discussion:
When you say title and slug are immune, do you mean that changes to those fields always trigger a rebuild or they do not trigger a rebuild?

Always trigger.


Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -318,8 +318,10 @@ def deps(self, lang):
         deps = []
         if self.default_lang in self.translated_to:
             deps.append(self.base_path)
+            deps.append(self.source_path)
         if lang != self.default_lang:
-            deps += [get_translation_candidate(self.config, self.base_path, lang)]
+            deps.append(get_translation_candidate(self.config, self.base_path, lang))
+            deps.append(get_translation_candidate(self.config, self.source_path, lang))
         return deps
 
     def compile(self, lang):
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -671,7 +671,7 @@ def remove_file(source):
 # slugify is copied from
 # http://code.activestate.com/recipes/
 # 577257-slugify-make-a-string-usable-in-a-url-or-filename/
-_slugify_strip_re = re.compile(r'[^\w\s-]')
+_slugify_strip_re = re.compile(r'[^+\w\s-]')
 _slugify_hyphenate_re = re.compile(r'[-\s]+')
 
 


Test patch:
Repository: getnikola/nikola

Issue:
Wrong exit code from command line parsing
[/data/home/hayen/repos/nikola-site]> /opt/nikola/bin/nikola unknown
[2014-10-27T09:55:38Z] ERROR: Nikola: Unknown command unknown
[/data/home/hayen/repos/nikola-site]> echo $?
0

That should be "1" or so. I am using commands with "&&" and expect
Nikola to abort, if there was a problem. Not happening in this case.



Discussion:

Resolving patch:
diff --git a/nikola/__main__.py b/nikola/__main__.py
--- a/nikola/__main__.py
+++ b/nikola/__main__.py
@@ -270,12 +270,12 @@ def run(self, cmd_args):
             args = ['version']
         if args[0] not in sub_cmds.keys():
             LOGGER.error("Unknown command {0}".format(args[0]))
-            return False
+            return 3
         if not isinstance(sub_cmds[args[0]], (Command, Help)):  # Is a doit command
             if not self.nikola.configured:
                 LOGGER.error("This command needs to run inside an "
                              "existing Nikola site.")
-                return False
+                return 3
 
         return super(DoitNikola, self).run(cmd_args)
 


Test patch:
Repository: getnikola/nikola

Issue:
Hyphenate breaks inline math
If you activate `HYPHENATE = True` in the `conf.py` inline math is broken because pyphen inserts `-` in the `<span>` elements.  This bug has taken some time before I found the source, because the html looks okay in Chrome and Firefoxes developer tools.
A test post (w/o metadata): 

```
Some inline :math:`\int_0^\infty \mathcal{A}(x) dx` and display math:

.. math::

    \int_0^\infty \mathcal{A}(x) dx
```

To fix this, the `hyphenate` method in [post.py](https://github.com/getnikola/nikola/blob/master/nikola/post.py) line 841 has to be changed. 

The problem is, it's not enough to check if the current `node` has a `math` class attribute because if the inline-math `<span>` is contained in a `<p>` for example, hyphenate will insert the hyphens in the `<p>` element including all of it's childs. I've made quick hack to prevent the execution of `pyphen` on elements that contains `<span>` childs with a `math` class:

```
def hyphenate(dom, lang):
    if pyphen is not None:
        hyphenator = pyphen.Pyphen(lang=lang)
        for tag in ('p', 'li', 'span'):
            for node in dom.xpath("//%s[not(parent::pre)]" % tag):
                math_found = False
                if node.getchildren():
                    for child in node.getchildren():
                        if child.tag == 'span' and child.get('class') and 'math' in child.get('class'):
                            math_found = True
                else:
                    if child.get('class') and 'math' in child.get('class'):
                        math_found = True
                if not math_found:
                    print(node.text, math_found)
                    insert_hyphens(node, hyphenator)
    return dom
```

This doesn't look nice and is also not very efficient, maybe it's enough to warn users that inline-math is broken when using hyphenate.



Discussion:
This hack looks good enough.  Make a Pull Request out of it.

But this prevents hyphenate also on running from the elements that surrounds the math `<span>`.


Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -843,7 +843,16 @@ def hyphenate(dom, lang):
         hyphenator = pyphen.Pyphen(lang=lang)
         for tag in ('p', 'li', 'span'):
             for node in dom.xpath("//%s[not(parent::pre)]" % tag):
-                insert_hyphens(node, hyphenator)
+                math_found = False
+                if node.getchildren():
+                    for child in node.getchildren():
+                        if child.tag == 'span' and 'math' in child.get('class', []):
+                            math_found = True
+                else:
+                    if child.get('class') and 'math' in child.get('class'):
+                        math_found = True
+                if not math_found:
+                    insert_hyphens(node, hyphenator)
     return dom
 
 


Test patch:
Repository: getnikola/nikola

Issue:
Broken headlines using typogrify (caused by insertion of `&nbsp;`)
Maybe we should prevent typogrify on running on h-elements because otherwise you headings won't wrap like you expect on mobile displays. I have created an [issue](https://github.com/mintchaos/typogrify/issues/40) with a more detailed description in the typogrify repo. This is not a real typogrify "bug", but we could implement a workaround in the [filters.py](https://github.com/getnikola/nikola/blob/master/nikola/filters.py) on line 163, because I don't think that the current behaviour is what most nikola users would expect.



Discussion:
Why not?

Can you do it yourself (`widont()` is the issue), or should someone else (me?) handle this?

No, my idea was to ask if someone has a good argument for not doing this change in the filter behaviour.

For mobiles, things like widow prevention are less important.  It will work out well.

We could disable `data = typo.widont(data)` completely or orperate all the h-tags out of the `data` argument. But this would be a lot of work for almost no benefit.

Nobody probably notices the widows disappearing.  We could even take that out…

This is open for discussion though.  If there are no objections by Sunday, `widont()` is going to hell.

:+1: 
I've made a pull request.


Resolving patch:
diff --git a/nikola/filters.py b/nikola/filters.py
--- a/nikola/filters.py
+++ b/nikola/filters.py
@@ -160,7 +160,8 @@ def typogrify(data):
         req_missing(['typogrify'], 'use the typogrify filter')
 
     data = typo.amp(data)
-    data = typo.widont(data)
+    # disabled because typogrify widow prevention caused broken headline wrapping, see issue #1465
+    # data = typo.widont(data)
     data = typo.smartypants(data)
     # Disabled because of typogrify bug where it breaks <title>
     # data = typo.caps(data)


Test patch:
Repository: getnikola/nikola

Issue:
github_deploy broken if no remote `gh-pages` branch
If you just started out with Nikola and your remote doesn't have a `gh-pages` branch yet (which will be the case for most people), then `github_deploy` crashes with:

```
ERROR: github_deploy: Failed GitHub deployment — command ['git', 'pull', 'origin', 'gh-pages:gh-pages'] returned 1
```

Since `git pull origin gh-pages:gh-pages` won't work if there's no remote `gh-pages` branch.  This is a big problem, because creating and pushing the blank branch is possible but nontrivial, and is one of the main automation "bonuses" of having a `nikola github_deploy` in the first place.



Discussion:
A workaround, for the time being, is to do the following (with a clean repo, of course!) with a new project before using `github_deploy`:

```
git checkout --orphan gh-pages
git rm -rf .
git commit --allow-empty -m 'first commit'
git push origin gh-pages:gh-pages
git checkout master
```

We'll just nuke that git pull or at least make it optional.

## 

Chris “Kwpolska” Warrick http://chriswarrick.com/
Sent from my Galaxy S3.
On Oct 30, 2014 9:22 PM, "john krauss" notifications@github.com wrote:

> A workaround, for the time being, is to do the following (with a clean
> repo, of course!) with a new project before using github_deploy:
> 
> git checkout --orphan gh-pages
> git rm -rf .
> git commit --allow-empty -m 'first commit'
> git push origin gh-pages:gh-pages
> git checkout master
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/getnikola/nikola/issues/1464#issuecomment-61163123.

Another possibility would be to `git pull ... || echo "Could not pull"` or somesuch, to ignore a non-zero exit from the pull (essentially using it when possible, but ignoring it otherwise.)


Resolving patch:
diff --git a/nikola/plugins/command/github_deploy.py b/nikola/plugins/command/github_deploy.py
--- a/nikola/plugins/command/github_deploy.py
+++ b/nikola/plugins/command/github_deploy.py
@@ -95,6 +95,9 @@ def _execute(self, command, args):
         self._remote_name = self.site.config.get(
             'GITHUB_REMOTE_NAME', 'origin'
         )
+        self._pull_before_commit = self.site.config.get(
+            'GITHUB_PULL_BEFORE_COMMIT', False
+        )
 
         self._ensure_git_repo()
 
@@ -135,13 +138,15 @@ def _commit_and_push(self):
         )
 
         commands = [
-            ['git', 'pull', '--rebase=false', remote, '%s:%s' % (deploy, deploy)],
             ['git', 'add', '-A'],
             ['git', 'commit', '-m', commit_message],
             ['git', 'push', '--force', remote, '%s:%s' % (deploy, deploy)],
             ['git', 'checkout', source],
         ]
 
+        if self._pull_before_commit:
+            commands.insert(0, ['git', 'pull', '--rebase=false', remote, '%s:%s' % (deploy, deploy)])
+
         for command in commands:
             self.logger.info("==> {0}".format(command))
             try:


Test patch:
Repository: getnikola/nikola

Issue:
Make indexes-related settings translatable
- [x] `INDEXES_TITLE`
- [x] `INDEXES_PAGES`

should be translatable.



Discussion:
Good idea!

(I'll do it tomorrow; most of the architecture is already there though.)

## 

Chris Warrick https://chriswarrick.com/
Sent from my Galaxy S3.
On Dec 21, 2014 9:49 PM, "felixfontein" notifications@github.com wrote:

> Good idea!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/getnikola/nikola/issues/1544#issuecomment-67784162.


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -443,7 +443,9 @@ def __init__(self, **config):
                                       'EXTRA_HEAD_DATA',
                                       'NAVIGATION_LINKS',
                                       'INDEX_READ_MORE_LINK',
-                                      'RSS_READ_MORE_LINK',)
+                                      'RSS_READ_MORE_LINK',
+                                      'INDEXES_TITLE',
+                                      'INDEXES_PAGES',)
 
         self._GLOBAL_CONTEXT_TRANSLATABLE = ('blog_author',
                                              'blog_title',
@@ -1484,7 +1486,7 @@ def generic_index_renderer(self, lang, posts, indexes_title, template_name, cont
         kw["tag_pages_are_indexes"] = self.config['TAG_PAGES_ARE_INDEXES']
         kw["index_display_post_count"] = self.config['INDEX_DISPLAY_POST_COUNT']
         kw["index_teasers"] = self.config['INDEX_TEASERS']
-        kw["indexes_pages"] = self.config['INDEXES_PAGES']
+        kw["indexes_pages"] = self.config['INDEXES_PAGES'](lang)
         kw["indexes_pages_main"] = self.config['INDEXES_PAGES_MAIN']
         kw["indexes_static"] = self.config['INDEXES_STATIC']
 
diff --git a/nikola/plugins/task/indexes.py b/nikola/plugins/task/indexes.py
--- a/nikola/plugins/task/indexes.py
+++ b/nikola/plugins/task/indexes.py
@@ -70,7 +70,7 @@ def page_path(i, num_pages):
             else:
                 filtered_posts = [x for x in posts if x.is_translation_available(lang)]
 
-            indexes_title = kw['indexes_title'] or kw['blog_title'](lang)
+            indexes_title = kw['indexes_title'](lang) or kw['blog_title'](lang)
 
             yield self.site.generic_index_renderer(lang, filtered_posts, indexes_title, template_name, {}, kw, 'render_indexes', page_link, page_path)
 


Test patch:
Repository: getnikola/nikola

Issue:
All RSS feeds contain invalid link to rss.xsl for subdirectory deployments
Spotted another 404 in the logs. `http://example.com/blog/rss.xml` contains `href="../assets/xml/rss.xsl"` which leads outside of the blog directory.  Same thing with other rss feeds in subdirectories.



Discussion:

Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -1017,7 +1017,8 @@ def generic_rss_renderer(self, lang, title, link, description, timeline, output_
         )
 
         if feed_url:
-            rss_obj.xsl_stylesheet_href = self.url_replacer(feed_url, "/assets/xml/rss.xsl")
+            absurl = '/' + feed_url[len(self.config['BASE_URL']):]
+            rss_obj.xsl_stylesheet_href = self.url_replacer(absurl, "/assets/xml/rss.xsl")
 
         items = []
 
diff --git a/nikola/plugins/task/rss.py b/nikola/plugins/task/rss.py
--- a/nikola/plugins/task/rss.py
+++ b/nikola/plugins/task/rss.py
@@ -51,6 +51,7 @@ def gen_tasks(self):
             "filters": self.site.config["FILTERS"],
             "blog_title": self.site.config["BLOG_TITLE"],
             "site_url": self.site.config["SITE_URL"],
+            "base_url": self.site.config["BASE_URL"],
             "blog_description": self.site.config["BLOG_DESCRIPTION"],
             "output_folder": self.site.config["OUTPUT_FOLDER"],
             "rss_teasers": self.site.config["RSS_TEASERS"],
diff --git a/nikola/plugins/task/tags.py b/nikola/plugins/task/tags.py
--- a/nikola/plugins/task/tags.py
+++ b/nikola/plugins/task/tags.py
@@ -58,6 +58,7 @@ def gen_tasks(self):
             "translations": self.site.config["TRANSLATIONS"],
             "blog_title": self.site.config["BLOG_TITLE"],
             "site_url": self.site.config["SITE_URL"],
+            "base_url": self.site.config["BASE_URL"],
             "messages": self.site.MESSAGES,
             "output_folder": self.site.config['OUTPUT_FOLDER'],
             "filters": self.site.config['FILTERS'],


Test patch:
Repository: getnikola/nikola

Issue:
reST titles broken?
Input:

``` rest
=====
hello
=====

some text
```

Output completely ignores the `hello` title.  It should be handled by _get_metadata_from_file and made the document title, but it gets ignored for some reason…  I don’t understand what the regexp is supposed to do though.

It does not appear in the final HTML, because we don’t use `pub.writer.parts['body_pre_docinfo']`.

cc @ralsina



Discussion:
I am leaving on vacation but I'll take a look in ~10 days

Have fun!

Basically, the regexp there matches things like 

```
Foobar
======
```

Not 

```
========
Foobar
========
```

I will try to expand it, or else document it.


Resolving patch:
diff --git a/nikola/post.py b/nikola/post.py
--- a/nikola/post.py
+++ b/nikola/post.py
@@ -786,6 +786,8 @@ def _get_metadata_from_file(meta_data):
     >>> g = _get_metadata_from_file
     >>> list(g([]).values())
     []
+    >>> str(g(["======","FooBar","======"])["title"])
+    'FooBar'
     >>> str(g(["FooBar","======"])["title"])
     'FooBar'
     >>> str(g(["#FooBar"])["title"])
@@ -820,6 +822,10 @@ def _get_metadata_from_file(meta_data):
         if 'title' not in meta:
             if re_rst_title.findall(line) and i > 0:
                 meta['title'] = meta_data[i - 1].strip()
+        if 'title' not in meta:
+            if (re_rst_title.findall(line) and i >= 0 and
+                    re_rst_title.findall(meta_data[i + 2])):
+                meta['title'] = meta_data[i + 1].strip()
         if 'title' not in meta:
             if re_md_title.findall(line):
                 meta['title'] = re_md_title.findall(line)[0]


Test patch:
Repository: getnikola/nikola

Issue:
Build fails with IPython 3.0
Trying to use ipython notebooks with the current dev version of IPython (3.0.0) fails building with some warnings etc. because the `nbformat` interface has changed a little:

```
...WARNING: UserWarning: .../ipython-dev/IPython/nbformat/current.py:19: IPython.nbformat.current is deprecated.

- use IPython.nbformat for read/write/validate public API
- use IPython.nbformat.vX directly to composing notebooks of a particular version
...
... WARNING: UserWarning: .../ipython-dev/IPython/nbformat/current.py:75: reads_json is deprecated, use reads
...
AttributeError: cells
```

This is fairly easily fixed and I will send a PR shortly.



Discussion:

Resolving patch:
diff --git a/nikola/plugins/compile/ipynb/__init__.py b/nikola/plugins/compile/ipynb/__init__.py
--- a/nikola/plugins/compile/ipynb/__init__.py
+++ b/nikola/plugins/compile/ipynb/__init__.py
@@ -31,8 +31,15 @@
 import os
 
 try:
+    import IPython
     from IPython.nbconvert.exporters import HTMLExporter
-    from IPython.nbformat import current as nbformat
+    if IPython.version_info[0] >= 3:     # API changed with 3.0.0
+        from IPython import nbformat
+        current_nbformat = nbformat.current_nbformat
+    else:
+        import IPython.nbformat.current as nbformat
+        current_nbformat = 'json'
+
     from IPython.config import Config
     flag = True
 except ImportError:
@@ -58,8 +65,7 @@ def compile_html(self, source, dest, is_two_file=True):
         exportHtml = HTMLExporter(config=c)
         with io.open(dest, "w+", encoding="utf8") as out_file:
             with io.open(source, "r", encoding="utf8") as in_file:
-                nb = in_file.read()
-                nb_json = nbformat.reads_json(nb)
+                nb_json = nbformat.read(in_file, current_nbformat)
             (body, resources) = exportHtml.from_notebook_node(nb_json)
             out_file.write(body)
 


Test patch: