internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: getpelican/pelican

Issue:
Feeds generate malformed urls (with double slash)
When setting a site url that ends with a slash (like http://feldboris.alwaysdata.net/blog/), pelican will generate url in feeds with double slash (like http://feldboris.alwaysdata.net/blog//feeds/all.atom.xml) just after site url. W3C feeds validator show a warning with these double slashs.

It's maybe the cause of a rss problem (my rss client show old feeds entries as unread) as post fields also contains double slash now.



Discussion:
well, the obvious solution seems to not add a / at the end of the 
siteurl.

Does it cause any problem to not have it?

I noticed the same problem and eventually realized that I just needed to omit the trailing slash. Perhaps we could make this more clear in the docs for the SITEURL setting, explaining that the value should not have a trailing slash.

We can even detect this in pelican and throw a warning + remove it?


Resolving patch:
diff --git a/pelican/settings.py b/pelican/settings.py
--- a/pelican/settings.py
+++ b/pelican/settings.py
@@ -128,9 +128,15 @@ def configure_settings(settings, default_settings=None, filename=None):
     else:
         logger.warn("LOCALE option doesn't contain a correct value")
 
-    # If SITEURL is defined but FEED_DOMAIN isn't, set FEED_DOMAIN = SITEURL
-    if ('SITEURL' in settings) and (not 'FEED_DOMAIN' in settings):
-        settings['FEED_DOMAIN'] = settings['SITEURL']
+    if ('SITEURL' in settings):
+        # If SITEURL has a trailing slash, remove it and provide a warning
+        siteurl = settings['SITEURL']
+        if (siteurl[len(siteurl) - 1:] == '/'):
+            settings['SITEURL'] = siteurl[:-1]
+            logger.warn("Removed extraneous trailing slash from SITEURL.")
+        # If SITEURL is defined but FEED_DOMAIN isn't, set FEED_DOMAIN = SITEURL
+        if not 'FEED_DOMAIN' in settings:
+            settings['FEED_DOMAIN'] = settings['SITEURL']
 
     # Warn if feeds are generated with both SITEURL & FEED_DOMAIN undefined
     if (('FEED' in settings) or ('FEED_RSS' in settings)) and (not 'FEED_DOMAIN' in settings):


Test patch:
diff --git a/tests/test_settings.py b/tests/test_settings.py
--- a/tests/test_settings.py
+++ b/tests/test_settings.py
@@ -36,6 +36,11 @@ def test_read_empty_settings(self):
     def test_configure_settings(self):
         """Manipulations to settings should be applied correctly."""
 
+        # SITEURL should not have a trailing slash
+        settings = {'SITEURL': 'http://blog.notmyidea.org/', 'LOCALE': ''}
+        configure_settings(settings)
+        self.assertEqual(settings['SITEURL'], 'http://blog.notmyidea.org')
+
         # FEED_DOMAIN, if undefined, should default to SITEURL
         settings = {'SITEURL': 'http://blog.notmyidea.org', 'LOCALE': ''}
         configure_settings(settings)
Repository: getpelican/pelican

Issue:
pelican-importer for Wordpress doesn't add tags
I tried both rst and markdown markups, and it doesn't work. I checked the wordpress.xml files, and they do have the tags.



Discussion:

Resolving patch:
diff --git a/pelican/tools/pelican_import.py b/pelican/tools/pelican_import.py
--- a/pelican/tools/pelican_import.py
+++ b/pelican/tools/pelican_import.py
@@ -34,7 +34,7 @@ def wp2fields(xml):
             categories = [cat.contents[0] for cat in item.fetch(domain='category')]
             # caturl = [cat['nicename'] for cat in item.fetch(domain='category')]
 
-            tags = [tag.contents[0].title() for tag in item.fetch(domain='tag', nicename=None)]
+            tags = [tag.contents[0] for tag in item.fetch(domain='post_tag')]
 
             yield (title, content, filename, date, author, categories, tags, "html")
 


Test patch:
Repository: getpelican/pelican

Issue:
Error installing on MS Windows
Installed:
- `virtualenv`
- `pip`

On a new environment, I did

```
pip install pelican
```

and I got:

```
installing scripts to build\bdist.win32\egg\EGG-INFO\scripts
running install_scripts
running build_scripts
error: file 'C:\Users\Oscar\my_new_env\build\pelican\bin\pelican.bat' does not exist
```

I downloaded the `.bat` file manually from the repository, and then it asked for three more `.bat` files.

It seems to be okay here on GitHub, but for some reason it isn't downloading an up-to-date version using `pip`



Discussion:

Resolving patch:
diff --git a/pelican/__init__.py b/pelican/__init__.py
--- a/pelican/__init__.py
+++ b/pelican/__init__.py
@@ -9,7 +9,7 @@
 from pelican.writers import Writer
 from pelican import log
 
-__version__ = "2.8.0"
+__version__ = "2.8.1"
 
 
 class Pelican(object):
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -3,7 +3,7 @@
 import sys
 import platform
 
-VERSION = "2.8.0" # find a better way to do so.
+VERSION = "2.8.1" # find a better way to do so.
 
 requires = ['feedgenerator', 'jinja2', 'pygments', 'docutils', 'pytz']
 if sys.version_info < (2,7):


Test patch:
Repository: getpelican/pelican

Issue:
Update documentation to cover hyprwave's page_name behavior
hyprwave provided an update to the page_name behavior to honor the TAG_URL, AUTHOR_URL, and CATEGORY_URL settings. This updates the documentation to include those updates.

Addresses issue #386 and #385



Discussion:

Resolving patch:
diff --git a/pelican/contents.py b/pelican/contents.py
--- a/pelican/contents.py
+++ b/pelican/contents.py
@@ -183,15 +183,23 @@ def __str__(self):
     def __unicode__(self):
         return self.name
 
-    def _from_settings(self, key):
+    def _from_settings(self, key, get_page_name=False):
+        """Returns URL information as defined in settings. 
+        When get_page_name=True returns URL without anything after {slug}
+        e.g. if in settings: CATEGORY_URL="cat/{slug}.html" this returns "cat/{slug}"
+        Useful for pagination."""
         setting = "%s_%s" % (self.__class__.__name__.upper(), key)
         value = self.settings[setting]
         if not isinstance(value, basestring):
             logger.warning(u'%s is set to %s' % (setting, value))
             return value
         else:
-            return unicode(value).format(**self.as_dict())
+            if get_page_name:
+                return unicode(value[:value.find('{slug}') + len('{slug}')]).format(**self.as_dict())
+            else:
+                return unicode(value).format(**self.as_dict())
 
+    page_name = property(functools.partial(_from_settings, key='URL', get_page_name=True))
     url = property(functools.partial(_from_settings, key='URL'))
     save_as = property(functools.partial(_from_settings, key='SAVE_AS'))
 
diff --git a/pelican/generators.py b/pelican/generators.py
--- a/pelican/generators.py
+++ b/pelican/generators.py
@@ -198,7 +198,7 @@ def generate_tags(self, write):
             write(tag.save_as, tag_template, self.context, tag=tag,
                 articles=articles, dates=dates,
                 paginated={'articles': articles, 'dates': dates},
-                page_name=u'tag/%s' % tag)
+                page_name=tag.page_name)
 
     def generate_categories(self, write):
         """Generate category pages."""
@@ -208,7 +208,7 @@ def generate_categories(self, write):
             write(cat.save_as, category_template, self.context,
                 category=cat, articles=articles, dates=dates,
                 paginated={'articles': articles, 'dates': dates},
-                page_name=u'category/%s' % cat)
+                page_name=cat.page_name)
 
     def generate_authors(self, write):
         """Generate Author pages."""
@@ -218,7 +218,7 @@ def generate_authors(self, write):
             write(aut.save_as, author_template, self.context,
                 author=aut, articles=articles, dates=dates,
                 paginated={'articles': articles, 'dates': dates},
-                page_name=u'author/%s' % aut)
+                page_name=aut.page_name)
 
     def generate_drafts(self, write):
         """Generate drafts pages."""


Test patch:
Repository: getpelican/pelican

Issue:
Autoreload stops on failure
When running pelican with --autoreload an error in rending the page not only stops the current render but completely quits pelican, forcing it to be reloaded. The render should fail but try again when the files are modified once more.

When working on a new template or fiddling with refactoring a website some people "twitch save" (an old finger memory habit of working with Microsoft word where you hit apple+s frequently) If there is a typo somewhere the whole thing dies and has to be manually restarted again, which seems to go against the whole idea of daemonizing the app.



Discussion:
+1. The way to do this is to add a try/catch around the pelican generate call, sounds quite simple, yet useful!


Resolving patch:
diff --git a/pelican/__init__.py b/pelican/__init__.py
--- a/pelican/__init__.py
+++ b/pelican/__init__.py
@@ -258,7 +258,13 @@ def main():
 
                     time.sleep(.5)  # sleep to avoid cpu load
                 except KeyboardInterrupt:
+                    logger.warning("Keyboard interrupt, quitting.")
                     break
+                except Exception, e:
+                    logger.warning(
+                        "Caught exception \"{}\". Reloading.".format(e)
+                    )
+                    continue
         else:
             pelican.run()
     except Exception, e:


Test patch:
Repository: getpelican/pelican

Issue:
Inconsistent Warning level on missing dependency
The levels of exception between pages and articles not being able to render it because of missing dependency is inconsistent
Articles it's only a warning, pages is an exception.

Example out of fresh install in temp env, build framework with quick start, added a page and article, ran make html 
WARNING: Could not process /folder/src/Announcements/Starting out.md
Missing dependencies for md
ERROR: Could not process /folder/src/pages/404.md
Missing dependencies for md



Discussion:
This is a bug :)


Resolving patch:
diff --git a/pelican/generators.py b/pelican/generators.py
--- a/pelican/generators.py
+++ b/pelican/generators.py
@@ -368,7 +368,7 @@ def generate_context(self):
             try:
                 content, metadata = read_file(f)
             except Exception, e:
-                logger.error(u'Could not process %s\n%s' % (f, str(e)))
+                logger.warning(u'Could not process %s\n%s' % (f, str(e)))
                 continue
             page = Page(content, metadata, settings=self.settings,
                         filename=f)


Test patch:
Repository: getpelican/pelican

Issue:
local variable 'metadata' referenced before assignment
First time trying to install and run Pelican. 
I installed Pelican with `pip install -e git://github.com/ametaireau/pelican#egg=pelican`

I am running `make html` which returns the error

`CRITICAL: local variable 'metadata' referenced before assignment`



Discussion:
Saw that too, the problem is from here (generators.py:398 and before).
It's probably easy to fix setting metadata to some null value before that, if that makes sense..

```
        try:
            content, metadata = read_file(f)
        except Exception, e:
            logger.warning(u'Could not process %s\n%s' % (f, str(e)))
            continue
        page = Page(content, metadata, settings=self.settings,
                    filename=f)
```

:+1: thank you.

Well haven't done anything yet ;)

Anyway running in debug mode I get this:

CRITICAL: local variable 'metadata' referenced before assignment
Traceback (most recent call last):
  File "/home/andrea/.local/bin/pelican", line 9, in <module>
    load_entry_point('pelican==3.0', 'console_scripts', 'pelican')()
  File "/home/andrea/pelican/pelican/**init**.py", line 296, in main
    pelican.run()
  File "/home/andrea/pelican/pelican/**init**.py", line 162, in run
    p.generate_context()
  File "/home/andrea/pelican/pelican/generators.py", line 398, in generate_context
    signals.pages_generate_context.send(self,metadata=metadata) 
UnboundLocalError: local variable 'metadata' referenced before assignment

I haven't had time to dig into this one too deep yet but the problem is rooted in this PR https://github.com/ametaireau/pelican/commit/65948c8137b244e95357b8b21a8db260fa12835f

When I get some more time I will work on resolving this, top priority.


Resolving patch:
diff --git a/pelican/generators.py b/pelican/generators.py
--- a/pelican/generators.py
+++ b/pelican/generators.py
@@ -382,6 +382,7 @@ def generate_context(self):
             except Exception, e:
                 logger.warning(u'Could not process %s\n%s' % (f, str(e)))
                 continue
+            signals.pages_generate_context.send(self, metadata=metadata )
             page = Page(content, metadata, settings=self.settings,
                         filename=f)
             if not is_valid_content(page, f):
@@ -395,7 +396,6 @@ def generate_context(self):
                                (repr(unicode.encode(page.status, 'utf-8')),
                                 repr(f)))
 
-        signals.pages_generate_context.send(self,metadata=metadata) 
         self.pages, self.translations = process_translations(all_pages)
         self.hidden_pages, self.hidden_translations = process_translations(hidden_pages)
 


Test patch:
Repository: getpelican/pelican

Issue:
Crash when DELETE_OUTPUT_DIRECTORY=True and output dir does not exist
If the output directory does not exist, and `DELETE_OUTPUT_DIRECTORY=True` in the settings file, pelican exits with an error:

```
CRITICAL: [Errno 2] No such file or directory: '/path/to/deploy'
```

I have a patch for this, and will submit a pull request.



Discussion:

Resolving patch:
diff --git a/pelican/utils.py b/pelican/utils.py
--- a/pelican/utils.py
+++ b/pelican/utils.py
@@ -97,6 +97,17 @@ def copy(path, source, destination, destination_path=None, overwrite=False):
 def clean_output_dir(path):
     """Remove all the files from the output directory"""
 
+    if not os.path.exists(path):
+        logger.debug("Directory already removed: %s" % path)
+        return
+
+    if not os.path.isdir(path):
+        try:
+            os.remove(path)
+        except Exception, e:
+            logger.error("Unable to delete file %s; %e" % path, e)
+        return
+
     # remove all the existing content from the output folder
     for filename in os.listdir(path):
         file = os.path.join(path, filename)


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -112,3 +112,16 @@ def test_clean_output_dir(self):
         self.assertTrue(os.path.isdir(test_directory))
         self.assertListEqual([], os.listdir(test_directory))
         shutil.rmtree(test_directory)
+
+    def test_clean_output_dir_not_there(self):
+        test_directory = os.path.join(os.path.dirname(__file__), 'does_not_exist')
+        utils.clean_output_dir(test_directory)
+        self.assertTrue(not os.path.exists(test_directory))
+
+    def test_clean_output_dir_is_file(self):
+        test_directory = os.path.join(os.path.dirname(__file__), 'this_is_a_file')
+        f = open(test_directory, 'w')
+        f.write('')
+        f.close()
+        utils.clean_output_dir(test_directory)
+        self.assertTrue(not os.path.exists(test_directory))
Repository: getpelican/pelican

Issue:
Crash when FILES_TO_COPY dest dir missing
If `FILES_TO_COPY` has a dest dir that is not in the root, for example:

```
FILES_TO_COPY = [
    ('internal/stuff.c', 'internal/stuff.c'),
    ]
```

and the named directory does not exist, pelican exits with an error:

```
CRITICAL: [Errno 2] No such file or directory: '/path/to/deploy/internal/stuff.c'
```

As a workaround, you can manually create the output directory, but this strategy won't work if you're using `DELETE_OUTPUT_FILES=True`.

I have a patch for this, and will submit a pull request.



Discussion:

Resolving patch:
diff --git a/pelican/utils.py b/pelican/utils.py
--- a/pelican/utils.py
+++ b/pelican/utils.py
@@ -90,6 +90,12 @@ def copy(path, source, destination, destination_path=None, overwrite=False):
                 logger.info('replacement of %s with %s' % (source_, destination_))
 
     elif os.path.isfile(source_):
+        try:
+            dest_dir = os.path.dirname(destination_)
+            if not os.path.exists(dest_dir):
+                os.makedirs(dest_dir)
+        except OSError, e:
+            logger.error("copying: %s -> %s: %r" % (source_, dest_dir, e))
         shutil.copy(source_, destination_)
         logger.info('copying %s to %s' % (source_, destination_))
 


Test patch:
Repository: getpelican/pelican

Issue:
Pagination fails when files don't have an extension
The pagination process assumes that files will have extensions (i.e. usually ".html"). This fails if the URL scheme for the site uses extension-less names. My settings.py has:

```
ARTICLE_URL = '{slug}'
ARTICLE_SAVE_AS = '{slug}'
AUTHOR_URL = 'author/{name}.html'
CATEGORY_URL = 'category/{name}'
CATEGORY_SAVE_AS = 'category/{name}'
TAG_URL = 'tag/{name}'
TAG_SAVE_AS = 'tag/{name}'
```

This means that, for example, the linux tag is listed at `http://example.com/tag/linux` -- note: no extension. Using commit  1580f7f (current master as of this writing), only the final page of tags is generated, because the same file is rewritten for each page because the name doesn't change.

I have a patch for this and will submit a PR.



Discussion:

Resolving patch:
diff --git a/pelican/writers.py b/pelican/writers.py
--- a/pelican/writers.py
+++ b/pelican/writers.py
@@ -148,9 +148,9 @@ def _write_file(template, localcontext, output_path, name):
                     paginators[key] = Paginator(object_list, len(object_list))
 
             # generated pages, and write
+            name_root, ext = os.path.splitext(name)
             for page_num in range(paginators.values()[0].num_pages):
                 paginated_localcontext = localcontext.copy()
-                paginated_name = name
                 for key in paginators.iterkeys():
                     paginator = paginators[key]
                     page = paginator.page(page_num + 1)
@@ -158,9 +158,10 @@ def _write_file(template, localcontext, output_path, name):
                             {'%s_paginator' % key: paginator,
                              '%s_page' % key: page})
                 if page_num > 0:
-                    ext = '.' + paginated_name.rsplit('.')[-1]
-                    paginated_name = paginated_name.replace(ext,
-                        '%s%s' % (page_num + 1, ext))
+                    paginated_name = '%s%s%s' % (
+                        name_root, page_num + 1, ext)
+                else:
+                    paginated_name = name
 
                 _write_file(template, paginated_localcontext, self.output_path,
                     paginated_name)


Test patch:
Repository: getpelican/pelican

Issue:
Getting an IndexError: list index out of range while importing from WP xml file
I'm trying to import this [xml file](https://gist.github.com/3173797) using pelican-import.

This is what I'm trying to do:

`pelican-import --wpfile programmingbeans.wordpress.2012-07-25.xml`

And here's the output:

``` python
output/about.rst
output/to-do-list.rst
output/status-as-of-feb-52011.rst
Traceback (most recent call last):
  File "/venv/bin/pelican-import", line 283, in <module>
    main(input_type, args.input, args.markup, args.output, dircat=args.dircat)
  File "/venv/bin/pelican-import", line 242, in main
    fields2pelican(fields, out_markup, output_path, dircat=dircat)
  File "/venv/bin/pelican-import", line 188, in fields2pelican
    for title, content, filename, date, author, categories, tags, in_markup in fields:
  File "/venv/bin/pelican-import", line 22, in wp2fields
    title = item.title.contents[0]
IndexError: list index out of range
```



Discussion:

Resolving patch:
diff --git a/pelican/tools/pelican_import.py b/pelican/tools/pelican_import.py
--- a/pelican/tools/pelican_import.py
+++ b/pelican/tools/pelican_import.py
@@ -14,10 +14,14 @@
 import subprocess
 import sys
 import time
+import logging
 
 from codecs import open
 
 from pelican.utils import slugify
+from pelican.log import init
+
+logger = logging.getLogger(__name__)
 
 
 def decode_wp_content(content, br=True):
@@ -114,7 +118,8 @@ def wp2fields(xml):
                 # Use HTMLParser due to issues with BeautifulSoup 3
                 title = HTMLParser().unescape(item.title.contents[0])
             except IndexError:
-                continue
+                title = 'No title [%s]' % item.find('post_name').string
+                logger.warn('Post "%s" is lacking a proper title' % title)
 
             content = item.find('encoded').string
             filename = item.find('post_name').string
@@ -492,6 +497,8 @@ def main():
     elif input_type == 'feed':
         fields = feed2fields(args.input)
 
+    init() # init logging
+
     fields2pelican(fields, args.markup, args.output,
                    dircat=args.dircat or False,
                    strip_raw=args.strip_raw or False,


Test patch:
Repository: getpelican/pelican

Issue:
pelican-import unable to import blog entries that contain characters that you can’t put in Windows file names.
Hi there!

I’m currently trying to import my old entries from Blogger (by converting the Blogger XML export results to Wordpress, and then using `pelican-import --wpfile`, but apparently `pelican-import` stops with the following error:

> Traceback (most recent call last):
>  File "C:\Python\old-blog-graveyard\Scripts\pelican-import-script.py", line 9, in <module>
>    load_entry_point('pelican==3.1.1', 'console_scripts', 'pelican-import')()
>  File "C:\Python\old-blog-graveyard\lib\site-packages\pelican\tools\pelican_import.py", line 331, in main
>    strip_raw=args.strip_raw or False)
>  File "C:\Python\old-blog-graveyard\lib\site-packages\pelican\tools\pelican_import.py", line 238, in fields2pelican
>    with open(html_filename, 'w', encoding='utf-8') as fp:
>  File "C:\Python\old-blog-graveyard\lib\codecs.py", line 881, in open
>     file = **builtin**.open(filename, mode, buffering)
> IOError: [Errno 22] invalid mode ('wb') or filename: u'output\What-the-hell-hit-me?.html'

I’ve received multiple errors like this, normally when the wp:post_name tag contains characters that are not valid in Windows: `< > : " / \ | ? *`

My workaround was, of course, to remove those entries manually using emacs & regexp, but I might want to suggest that you filter out or sanitize filenames before writing them to disk, especially on Windows systems (although I’m sure Mac OS X and Linux have their own conventions on what constitutes invalid characters in a filename).



Discussion:

Resolving patch:
diff --git a/pelican/tools/pelican_import.py b/pelican/tools/pelican_import.py
--- a/pelican/tools/pelican_import.py
+++ b/pelican/tools/pelican_import.py
@@ -376,6 +376,15 @@ def fields2pelican(fields, out_markup, output_path, dircat=False, strip_raw=Fals
 
         filename = os.path.basename(filename)
 
+        # Enforce filename restrictions for various filesystems at once; see
+        # http://en.wikipedia.org/wiki/Filename#Reserved_characters_and_words
+        # we do not need to filter words because an extension will be appended
+        filename = re.sub(r'[<>:"/\\|?*^% ]', '-', filename) # invalid chars
+        filename = filename.lstrip('.') # should not start with a dot
+        if not filename:
+            filename = '_'
+        filename = filename[:249] # allow for 5 extra characters
+
         # option to put files in directories with categories names
         if dircat and (len(categories) > 0):
             catname = slugify(categories[0])


Test patch: