internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getnikola/nikola

Issue:
Complete _WINDOWS_LOCALE_GUESSES
Currently, `_WINDOWS_LOCALE_GUESSES` has only 20 of 30+ locales. It should be updated to try to equate it with `TRANSLATIONS`.



Discussion:
@ccanepa, where is this information? There was a link, but the information is gone. Any hints?

You could try with https://msdn.microsoft.com/en-us/goglobal/bb964664.aspx

This is a list, but for Windows XP: https://msdn.microsoft.com/en-us/goglobal/bb895996

@sukiletxe would you mind doing that, or should I try?

Oops, I sent you the wrong page. Anyways, the list you sent seems to be 
pretty complete. I'll try fixing this myself.

Just one thing: it seems that some locales have their own dialects (such 
as Arabic), and no generic locale. Nontheless, I will try with just the 
generic name, as it has worked for English.


Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -129,7 +129,7 @@
         ('ja', '!jp'): 'Japanese',
         'ko': 'Korean',
         'lt': 'Lithuanian',
-        'nb': 'Norwegian Bokmål',
+        'nb': 'Norwegian (Bokmål)',
         'nl': 'Dutch',
         'pa': 'Punjabi',
         'pl': 'Polish',
@@ -148,26 +148,47 @@
     },
     '_WINDOWS_LOCALE_GUESSES': {
         # TODO incomplete
-        # some languages may need that the appropiate Microsoft Language Pack be instaled.
+        # some languages may need that the appropriate Microsoft Language Pack be installed.
+        "ar": "Arabic",
+        "az": "Azeri (Latin)",
         "bg": "Bulgarian",
+        "bs": "Bosnian",
         "ca": "Catalan",
+        "cs": "Czech",
+        "da": "Danish",
         "de": "German",
         "el": "Greek",
         "en": "English",
-        "eo": "Esperanto",
+        # "eo": "Esperanto", # Not available
         "es": "Spanish",
-        "fa": "Farsi",  # Persian
+        "et": "Estonian",
+        "eu": "Basque",
+        "fa": "Persian",  # Persian
+        "fi": "Finnish",
         "fr": "French",
+        "gl": "Galician",
+        "hi": "Hindi",
         "hr": "Croatian",
         "hu": "Hungarian",
+        "id": "Indonesian",
         "it": "Italian",
-        "jp": "Japanese",
+        "ja": "Japanese",
+        "ko": "Korean",
+        "nb": "Norwegian",  # Not Bokmål, as Windows doesn't find it for unknown reasons.
         "nl": "Dutch",
+        "pa": "Punjabi",
         "pl": "Polish",
+        "pt": "Portuguese_Portugal",
         "pt_br": "Portuguese_Brazil",
         "ru": "Russian",
-        "sl_si": "Slovenian",
-        "tr_tr": "Turkish",
+        "sk": "Slovak",
+        "sl": "Slovenian",
+        "sr": "Serbian",
+        "sr_latin": "Serbian (Latin)",
+        "sv": "Swedish",
+        "tr": "Turkish",
+        "uk": "Ukrainian",
+        "ur": "Urdu",
         "zh_cn": "Chinese_China",  # Chinese (Simplified)
     },
     '_TRANSLATIONS_WITH_COUNTRY_SPECIFIERS': {


Test patch:
Repository: getnikola/nikola

Issue:
inability to uninstall plugins
Attempt to uninstall plugin gives:

```
nikola plugin -r localsearch
[2016-06-03T09:30:46Z] ERROR: plugin: Unknown plugin: localsearch
```

However, plugin is installed:

```
tree plugins/
plugins/
├── __init__.py
└── localsearch
    ├── conf.py.sample
    ├── localsearch
    │   ├── files
    │   │   ├── assets
    │   │   │   ├── css
    │   │   │   │   ├── img
    │   │   │   │   │   ├── loader.gif
    │   │   │   │   │   └── search.png
    │   │   │   │   └── tipuesearch.css
    │   │   │   └── js
    │   │   │       ├── tipuesearch.js
    │   │   │       ├── tipuesearch.min.js
    │   │   │       └── tipuesearch_set.js
    │   │   └── tipue_search.html
    │   ├── __init__.py
    │   └── __pycache__
    │       └── __init__.cpython-35.pyc
    ├── localsearch.plugin
    ├── MIT-LICENSE.txt
    └── README.md
```

Version of Nikola is 7.7.8.



Discussion:
The directory is called `localsearch` but the plugin's name is `local_search`. Note the `_`. 

@punchagan 

```
nikola plugin -l  | grep search
[2016-06-03T12:13:51Z] INFO: requests.packages.urllib3.connectionpool: Starting new HTTPS connection (1): plugins.getnikola.com
localsearch
```

Moreover, what about this:

```
[gour@atmarama ~]$ vf workon org
[gour@atmarama ~/prj/vf/org]$ (org) nikola plugin -i forms
[2016-06-03T12:12:15Z] INFO: requests.packages.urllib3.connectionpool: Starting new HTTPS connection (1): plugins.getnikola.com
[2016-06-03T12:12:15Z] INFO: plugin: Downloading 'https://plugins.getnikola.com/v7/forms.zip'
[2016-06-03T12:12:15Z] INFO: requests.packages.urllib3.connectionpool: Starting new HTTPS connection (1): plugins.getnikola.com
[2016-06-03T12:12:16Z] INFO: plugin: Extracting: forms into plugins/
[gour@atmarama ~/prj/vf/org]$ (org) nikola plugin -r forms
[2016-06-03T12:12:38Z] ERROR: plugin: Unknown plugin: forms
```

Ok, so deleting plugins is buggy. I'll fix it.


Resolving patch:
diff --git a/nikola/plugins/command/plugin.py b/nikola/plugins/command/plugin.py
--- a/nikola/plugins/command/plugin.py
+++ b/nikola/plugins/command/plugin.py
@@ -296,12 +296,15 @@ def do_install(self, url, name, show_install_notes=True):
     def do_uninstall(self, name):
         """Uninstall a plugin."""
         for plugin in self.site.plugin_manager.getAllPlugins():  # FIXME: this is repeated thrice
-            p = plugin.path
-            if os.path.isdir(p):
-                p = p + os.sep
-            else:
-                p = os.path.dirname(p)
             if name == plugin.name:  # Uninstall this one
+                p = plugin.path
+                if os.path.isdir(p):
+                    # Plugins that have a package in them need to delete parent
+                    # Issue #2356
+                    p = p + os.sep
+                    p = os.path.abspath(os.path.join(p, os.pardir))
+                else:
+                    p = os.path.dirname(p)
                 LOGGER.warning('About to uninstall plugin: {0}'.format(name))
                 LOGGER.warning('This will delete {0}'.format(p))
                 sure = utils.ask_yesno('Are you sure?')


Test patch:
Repository: getnikola/nikola

Issue:
output of 'nikola auto' not visible with light themes
I use solarized-light theme in my(gnome) terminal and when I e.g. run 'nikola build', most of the output is not clearly visible - see the shot: http://pasteboard.co/1pIJ9kw9.png, so wonder if something can be done to make it more ui-friendly for light color themes?



Discussion:
Use a black theme. Will solve so many problems.

> Use a black theme. Will solve so many problems.

I was using dark themes, but found out it's not good for my eyes...and everything else does not matter in comparison. ;)

Ok, so, Nikola can not know what the color scheme you are using is. Right?

So, going to add an option to deactivate color output globally, and then you can use it.


Resolving patch:
diff --git a/nikola/__main__.py b/nikola/__main__.py
--- a/nikola/__main__.py
+++ b/nikola/__main__.py
@@ -68,7 +68,7 @@
 def main(args=None):
     """Run Nikola."""
     colorful = False
-    if sys.stderr.isatty() and os.name != 'nt':
+    if sys.stderr.isatty() and os.name != 'nt' and os.getenv('NIKOLA_MONO') is None:
         colorful = True
 
     ColorfulStderrHandler._colorful = colorful


Test patch:
Repository: getnikola/nikola

Issue:
Multiple copies of a plugin are activated
When you have the same plugin installed in more than one place, instead of using the "most local" one, both are activated, which can cause conflicts.

Proposed solution: if there are more than 1 copy of the plugin, deactivate those that are less local.



Discussion:

Resolving patch:
diff --git a/nikola/nikola.py b/nikola/nikola.py
--- a/nikola/nikola.py
+++ b/nikola/nikola.py
@@ -888,8 +888,8 @@ def init_plugins(self, commands_only=False, load_all=False):
         if sys.version_info[0] == 3:
             self._plugin_places = [
                 resource_filename('nikola', 'plugins'),
-                os.path.join(os.getcwd(), 'plugins'),
                 os.path.expanduser('~/.nikola/plugins'),
+                os.path.join(os.getcwd(), 'plugins'),
             ] + [path for path in extra_plugins_dirs if path]
         else:
             self._plugin_places = [
@@ -930,6 +930,27 @@ def init_plugins(self, commands_only=False, load_all=False):
                         bad_candidates.add(p)
                         utils.LOGGER.debug('Not loading compiler extension {}', p[-1].name)
             self.plugin_manager._candidates = list(set(self.plugin_manager._candidates) - bad_candidates)
+
+        # Find repeated plugins and discard the less local copy
+        def plugin_position_in_places(plugin):
+            # plugin here is a tuple:
+            # (path to the .plugin file, path to plugin module w/o .py, plugin metadata)
+            for i, place in enumerate(self._plugin_places):
+                if plugin[0].startswith(place):
+                    return i
+
+        plugin_dict = defaultdict(list)
+        for data in self.plugin_manager._candidates:
+            plugin_dict[data[2].name].append(data)
+        self.plugin_manager._candidates = []
+        for name, plugins in plugin_dict.items():
+            if len(plugins) > 1:
+                # Sort by locality
+                plugins.sort(key=plugin_position_in_places)
+                utils.LOGGER.debug("Plugin {} exists in multiple places, using {}".format(
+                    plugins[-1][2].name, plugins[-1][0]))
+            self.plugin_manager._candidates.append(plugins[-1])
+
         self.plugin_manager.loadPlugins()
 
         self._activate_plugins_of_category("SignalHandler")


Test patch:
Repository: getnikola/nikola

Issue:
Arbitrary rst substitutions
Since we have transforms now, it should be easy to do something generic, like this:

`|bzr-revno|` that would be replaced by the output of `bzr revno` using a configuration like

``` py
def bzr_revno():
    return subprocess.check_output(["bzr", "revno"])

RST_SUBSTITUTIONS = {
    'bzr_revno': bzr_revno
}
```



Discussion:
Now I don't remember how this would have worked anymore :-D

just a random guess based on the original post and the rest plugin: http://docutils.sourceforge.net/docs/ref/transforms.html + self.site.rst_transforms?

Let’s scrap this and use shortcodes instead.

Shortcodes are horrible in rst because of quoting and random replacements
docutils does.

For example, anything that looks like a URL will become a link before
shortcodes get to it.

On Fri, Apr 29, 2016 at 8:31 AM Chris Warrick notifications@github.com
wrote:

> Let’s scrap this and use shortcodes instead.
> 
> —
> You are receiving this because you were assigned.
> Reply to this email directly or view it on GitHub
> https://github.com/getnikola/nikola/issues/1854#issuecomment-215688966

Well then, I’ll reopen and you might implement it if you ever feel like it.

Maybe a middle ground is a role that passes things through so you can use shortcodes without breaking stuff.


Resolving patch:
diff --git a/nikola/plugins/compile/rest/__init__.py b/nikola/plugins/compile/rest/__init__.py
--- a/nikola/plugins/compile/rest/__init__.py
+++ b/nikola/plugins/compile/rest/__init__.py
@@ -37,6 +37,7 @@
 import docutils.readers.standalone
 import docutils.writers.html4css1
 import docutils.parsers.rst.directives
+from docutils.parsers.rst import roles
 
 from nikola.plugin_categories import PageCompiler
 from nikola.utils import unicode_str, get_logger, makedirs, write_metadata, STDERR_HANDLER
@@ -193,6 +194,16 @@ def new_document(self):
         return document
 
 
+def shortcode_role(name, rawtext, text, lineno, inliner,
+                   options={}, content=[]):
+    """A shortcode role that passes through raw inline HTML."""
+    return [docutils.nodes.raw('', text, format='html')], []
+
+roles.register_canonical_role('raw-html', shortcode_role)
+roles.register_canonical_role('html', shortcode_role)
+roles.register_canonical_role('sc', shortcode_role)
+
+
 def add_node(node, visit_function=None, depart_function=None):
     """Register a Docutils node class.
 


Test patch:
Repository: getnikola/nikola

Issue:
new_post -2 -i could (should?) copy a file instead of read it.
I write my posts in MS Word, and have recently started trying to use the .docx files directly with Nikola.  Nikola supports compiling these documents out of the box, which is nice.  However, I would like to streamline the process further by using Nikola's built in import file functionality.

Currently, it attempts to read the file contents and pass it to the compiler's create post function.  While this works for most file types, it does not work for docx's binary file type (zipped xml) as it attempts to read the file as utf-8.

Because file types like docx have to use two file mode anyways, it would be convenient to copy the file instead of trying to read it when -2 is passed to new_post.



Discussion:
Sounds good.

:+1: from me. I’ll work on this later.


Resolving patch:
diff --git a/nikola/plugins/command/new_post.py b/nikola/plugins/command/new_post.py
--- a/nikola/plugins/command/new_post.py
+++ b/nikola/plugins/command/new_post.py
@@ -29,10 +29,11 @@
 from __future__ import unicode_literals, print_function
 import io
 import datetime
+import operator
 import os
-import sys
+import shutil
 import subprocess
-import operator
+import sys
 
 from blinker import signal
 import dateutil.tz
@@ -366,17 +367,22 @@ def _execute(self, options, args):
             onefile = False
             LOGGER.warn('This compiler does not support one-file posts.')
 
-        if import_file:
+        if onefile and import_file:
             with io.open(import_file, 'r', encoding='utf-8') as fh:
                 content = fh.read()
-        else:
+        elif not import_file:
             if is_page:
                 content = self.site.MESSAGES[self.site.default_lang]["Write your page here."]
             else:
                 content = self.site.MESSAGES[self.site.default_lang]["Write your post here."]
-        compiler_plugin.create_post(
-            txt_path, content=content, onefile=onefile, title=title,
-            slug=slug, date=date, tags=tags, is_page=is_page, **metadata)
+
+        if (not onefile) and import_file:
+            # Two-file posts are copied  on import (Issue #2380)
+            shutil.copy(import_file, txt_path)
+        else:
+            compiler_plugin.create_post(
+                txt_path, content=content, onefile=onefile, title=title,
+                slug=slug, date=date, tags=tags, is_page=is_page, **metadata)
 
         event = dict(path=txt_path)
 


Test patch:
Repository: getnikola/nikola

Issue:
Moving listings don’t rebuild the sitemap
```
LISTINGS_FOLDERS = { "projects/snippets": "projects/snippets" }

touch projects/snippets/example/test.1
nikola build
mv projects/snippets/example/test.1 projects/snippets/example/test.2
nikola build
nikola check --clean-files
nikola check -l
```

Broken link /projects/snippets/example/test.1 found in sitemap.xml. Projects/snippets/example/test.2 is not in the sitemap.



Discussion:
I think the problem is not what you think, and/or I can't reproduce this correctly.
- If I move a file inside listings, I do get the site rebuilt, and the new output file appears in sitemap.
- The old file _still_ appears in sitemap, but that's ok, because it's still in output and sitemap reflects reality
- If you do a --clean-files then yes, the old file is in sitemap but not there in reality, and rebuilding the site will do nothing.

So, maybe --clean-files should offer to rebuild the sitemap and/or mark the sitemap as dirty somehow so a rebuild will recreate it?


Resolving patch:
diff --git a/nikola/plugins/command/check.py b/nikola/plugins/command/check.py
--- a/nikola/plugins/command/check.py
+++ b/nikola/plugins/command/check.py
@@ -417,6 +417,8 @@ def clean_files(self):
             self.logger.debug('removed: {0}'.format(f))
             os.unlink(f)
 
+        warn_flag = bool(only_on_output)
+
         # Find empty directories and remove them
         output_folder = self.site.config['OUTPUT_FOLDER']
         all_dirs = []
@@ -427,6 +429,11 @@ def clean_files(self):
             try:
                 os.rmdir(d)
                 self.logger.debug('removed: {0}/'.format(d))
+                warn_flag = True
             except OSError:
                 pass
+
+        if warn_flag:
+            self.logger.warn('Some files or directories have been removed, your site may need rebuilding')
+
         return True


Test patch:
Repository: getnikola/nikola

Issue:
admonitions (warning, attention, hint, ...) not localized
(reST?) Admonitions are not localized in Nikola. The Docutils localization strings are already present in **docutils/languages/de.py**, but changing DEFAULT_LANG to "de" in conf.py does not use these localized strings when using the following (reST) code:

```
.. attention:: Headline

        Contents
```

The heading should be "Achtung!" but instead it is "Attention!"



Discussion:
there is a workaround if the whole blog only contains one language:

```
cat ~/.docutils
[general]
language_code=de
```

Looks like we are not passing language_code to docutils, should not be hard to fix, taking a look now.


Resolving patch:
diff --git a/nikola/plugins/compile/rest/__init__.py b/nikola/plugins/compile/rest/__init__.py
--- a/nikola/plugins/compile/rest/__init__.py
+++ b/nikola/plugins/compile/rest/__init__.py
@@ -40,7 +40,14 @@
 from docutils.parsers.rst import roles
 
 from nikola.plugin_categories import PageCompiler
-from nikola.utils import unicode_str, get_logger, makedirs, write_metadata, STDERR_HANDLER
+from nikola.utils import (
+    unicode_str,
+    get_logger,
+    makedirs,
+    write_metadata,
+    STDERR_HANDLER,
+    LocaleBorg
+)
 from nikola.shortcodes import apply_shortcodes
 
 
@@ -63,16 +70,19 @@ def compile_html_string(self, data, source_path=None, is_two_file=True):
             add_ln = len(m_data.splitlines()) + 1
 
         default_template_path = os.path.join(os.path.dirname(__file__), 'template.txt')
+        settings_overrides = {
+            'initial_header_level': 1,
+            'record_dependencies': True,
+            'stylesheet_path': None,
+            'link_stylesheet': True,
+            'syntax_highlight': 'short',
+            'math_output': 'mathjax',
+            'template': default_template_path,
+            'language_code': LocaleBorg().current_lang,
+        }
+
         output, error_level, deps = rst2html(
-            data, settings_overrides={
-                'initial_header_level': 1,
-                'record_dependencies': True,
-                'stylesheet_path': None,
-                'link_stylesheet': True,
-                'syntax_highlight': 'short',
-                'math_output': 'mathjax',
-                'template': default_template_path,
-            }, logger=self.logger, source_path=source_path, l_add_ln=add_ln, transforms=self.site.rst_transforms)
+            data, settings_overrides=settings_overrides, logger=self.logger, source_path=source_path, l_add_ln=add_ln, transforms=self.site.rst_transforms)
         if not isinstance(output, unicode_str):
             # To prevent some weird bugs here or there.
             # Original issue: empty files.  `output` became a bytestring.


Test patch:
Repository: getnikola/nikola

Issue:
Listings should always copy source files
If you have a source code listing, you want people to be able to download it anyway. Let’s ignore `COPY_SOURCES` for listings.



Discussion:

Resolving patch:
diff --git a/nikola/plugins/task/listings.py b/nikola/plugins/task/listings.py
--- a/nikola/plugins/task/listings.py
+++ b/nikola/plugins/task/listings.py
@@ -148,7 +148,7 @@ def render_listing(in_name, out_name, input_folder, output_folder, folders=[], f
                         os.path.join(
                             self.kw['output_folder'],
                             output_folder))))
-            if self.site.config['COPY_SOURCES'] and in_name:
+            if in_name:
                 source_link = permalink[:-5]  # remove '.html'
             else:
                 source_link = None
@@ -241,19 +241,19 @@ def render_listing(in_name, out_name, input_folder, output_folder, folders=[], f
                         'uptodate': [utils.config_changed(uptodate, 'nikola.plugins.task.listings:source')],
                         'clean': True,
                     }, self.kw["filters"])
-                    if self.site.config['COPY_SOURCES']:
-                        rel_name = os.path.join(rel_path, f)
-                        rel_output_name = os.path.join(output_folder, rel_path, f)
-                        self.register_output_name(input_folder, rel_name, rel_output_name)
-                        out_name = os.path.join(self.kw['output_folder'], rel_output_name)
-                        yield utils.apply_filters({
-                            'basename': self.name,
-                            'name': out_name,
-                            'file_dep': [in_name],
-                            'targets': [out_name],
-                            'actions': [(utils.copy_file, [in_name, out_name])],
-                            'clean': True,
-                        }, self.kw["filters"])
+
+                    rel_name = os.path.join(rel_path, f)
+                    rel_output_name = os.path.join(output_folder, rel_path, f)
+                    self.register_output_name(input_folder, rel_name, rel_output_name)
+                    out_name = os.path.join(self.kw['output_folder'], rel_output_name)
+                    yield utils.apply_filters({
+                        'basename': self.name,
+                        'name': out_name,
+                        'file_dep': [in_name],
+                        'targets': [out_name],
+                        'actions': [(utils.copy_file, [in_name, out_name])],
+                        'clean': True,
+                    }, self.kw["filters"])
 
     def listing_path(self, namep, lang):
         """A link to a listing.


Test patch: