spf13/hugo — pull request #451 by natefinch (2014-08-29 19:42:59+00:00)
Title: Fix CreatePages

This fixes #450.

There were three problems with CreatePages:

1.) We were creating a new goroutine for every page.

This is a problem in that if you have a ton of pages, that's a ton of goroutines. It's not really useful to have more than a few goroutines at a time, and lots can actually make your code much slower, and, evidently, crash.

2.) We were calling s.Pages = append(s.Pages, page) inside each goroutine.

This is a problem because append is not thread safe. Sometimes it returns a new slice with a larger capacity, when the original slice isn't large enough. This can cause problems if two goroutines do this at the same time.

3.) We were throwing away the errors from the page rendering.
Hopefully it's obvious why this is bad :)

The solution for 1 is to use a limited number of workers (I chose 2*GOMAXPROCS as a nice guess).
The solution for 2 is to serialize access to s.Pages, which I did by doing it in a single collator goroutine.
The solution for 3 is to have the workers send their errors to the collator, who then wraps them up and sends the result back to the main goroutine.