M-Lab/ip_addr.dat: sync
M-Lab/redir_table.sh: sync
www/header.html: sync with master

This diff minimizes the differences between next and master
such that it's a piece of cake to merge next into master.
net/poller.py: make sure we break loop when no I/O is pending

This has always been the behavior of Neubot poller, and, FWIW it
is also the behavior of asyncore.

After the stdlib-sched patch merge, the behavior has changed and has
become Twisted-like, i.e. the poller waits when there is no pending
I/O instead of exiting from the loop.  This wasn't done by purpose and
was just a side-effect of merging the stdlib-sched patch.

This commit modifies the poller such that it returns to the old
behavior, which is what the rest of Neubot expects.
runner_core.py: safely run callback()

This commit wraps callback() invokation with the proper try..except
clauses, to be sure that an error in callback() does not stop the
processing of the runner queue.

While there, s/conf/ctx/, because the object that is passed around
is indeed more a context than configuration.

Finally, if the ctx is not None, pass it to the callback so that
it can use it.

Changes either needed by or inspired by the new Win32 updater,
which has a component that is executed by the runner.

Checked that there are no obvious breakages by running this module
from command line.

No pylint or 2to3 changes.
rootdir.py: s/CANDIDATE/ROOTDIR/g

I want to import CANDIDATE, but that name is not meaningful,
and so let's rename it ROOTDIR.

No pylint or 2to3 changes.

Checked with grep -Rn that no file in neubot/ or regress/ was
using CANDIDATE.
api_server.py http_server.py poller.py: forward compatibility

In future Neubot will use a flat hierarchy and will not use
subdirectories, because that is simpler to manage.

In preparation for that, here are some wrappers that introduce
pieces of the flat hierarchy.
runner_core.py: print traceback not just exception

No pylint or 2to3 complaints.

Checked that there are no breakages running the script
from command line.
Merge cleanups for 0.4.11
Win32/setup.py: do not copy MacOSX special files
Merge branch 'next'
browser.py: use default URI if no URI is specified
config.py: add win32_updater setting

Initially I won't enable by default the updater, because it is
still beta software, but I will provide this setting to allow
people to test it.
add runner_dload.py: runner-based downloader

This runner-based downloader will be used by the windows updater,
which downloads updates in the same process context of neubot.

No new pylint error, clean 2to3, tested using the attached main()
function against various websites.
add updater_win32.py: automatic updates for windows

The toplevel module is updater_win32.py and depends on two
ancillary modules:

updater_runner.py
  This module contains the portable portion of code of
  updater_win32.py, for testability.

updater_utils.py
  Common function that will be shared with MacOS auto
  updater in future.

There are some pylint warnings in updater_runner.py and they
mainly depend TODO and XXX comments in the code.  Indeed, the
current version of this module is not as robust as I would
like, and I have added comments to remember what improvements
are due.

The 2to3 of updater_win32.py is not clean, but the code should
nonetheless work with python3 because I'm distinguishing the
case of Python3 and the case of Python2 when I import the winreg
module of standard library.
add background_win32.py: run neubot in background in windows

The background_win32.py delegates work to background_api.py (which
takes care of the API) and background_rendezvous.py (which runs auto
rendezvous).  This has been done for portability, because other OSs
will use the api and rendezvous pieces in the future.

There is a pylint warning in background_rendezvous.py, where I have
noticed that certain unexpected errors may stop automatic rendezvous
from running periodically.

No 2to3 issues.

Briefly tested background_api.py and background_rendezvous.py using
the command line on UNIX.
add main_win32.py: new main() for windows

The main module is main_win32.py, but there are also other
modules:

utils_ctl.py
  Contains shared helpers for controlling the daemon.

utils_module.py
  Contains code to run a given module by name.

I needed to add main() to speedtest/__init__.py so that the
speedtest command works on Windows too.

No complaints from pylint.

Instead 2to3 reminds me that utils_ctl.py is not portable
to python3 without (small) modifications.

I have guaranteed that all the modules that are listed in the
manual page still work with the new main.  There are no guarantees
for other modules.
Win32: use main_win32.py instead of main/__init__.py

Basically, create Win32-only copies of neubot and neubotw, and
remove bin/neubotw, which is not used anymore.

As a bonus, I've noticed that I was adding '.' to sys.path under
windows with no good reason, so I've removed the code that does
that, for the sake of correctness.
Win32: neubot.nsi setup.py: adapt to Win32 autoupdates

Hunk per hunk comment of this patch.

neubot.nsi
``````````

1. if running in uninstaller generation mode, write the uninstaller
   generator;

2. in normal mode, install everything on versiondir, including the
   uninstaller (see below);

3. if running in uninstaller generation mode, put the uninstaller
   in dist, so 2. will pick it;

4. the new style installs everything in versiondir, so there are no
   conflicts with old versions.  Still we need to check whether an
   old version is installed in LOCALAPPDATA and remove it;

5. do not write uninstaller in normal mode, because uninstaller
   generation mode already writes it into dist/;

6. we do not create anymore a shortcut in startup dir, because from
   python is easier to write the registry;

7. use ``neubot browser`` to start browser because ``neubot``
   alone now only prints usage on windows;

8. write the registry so neubot is run when the user logs into the
   system;

9. the uninstaller section is active only in uninstaller generation
   mode;

10. remove all LOCALAPPDATA\Neubot so we remove the current version
    plus any older version files;

11. we do not put a link in startup anymore, instead we need to
    remove the new registry key added in step 8.

Win32/setup.py
``````````````

12. run nsis in uninstaller generation mode, then run the uninstaller
    generator to generate ``dist/uninstall.exe``;

13. create an auto-updater tarball using the content of ``dist``.
updater_win32.py: Remember to close open handles before Popen()

Allows to access again the Web User Interface and API after an
automatic update.  Without this flag, the new process inherits the
listening handle, even if it does not know how to access it.
Interestingly, it successfully bind()s to 127.0.0.1:9774, but the
inherited handle "steals" new connections.  So every attempt to
reach 127.0.0.1:9774 eventually triggers a timeout.  The confusing
part is that `netstat -nao` tells you that the inherited handle
is bound by the old PID, not by the new one.