Work in progress porting from SQLite to MySQL.

- Basic fetch queries should work but that path will have a lot of debug
  output that will want deleting.

- Stats and other queries not ported at all

- Connection variables currently hardcoded and should be a config

- This branch started out as an attempt to do transactionless SQLite
  version with atomic operations so there may be cruft from that to
  clean up.

Change-Id: If4b2c2518f296a89a409a3a5328de5fc88aeac3b
Merge "Work in progress porting from SQLite to MySQL." into rt_testing
Port the round-trip test server to MySQL.
- Also ported importJson to mysql.
- Commit results and stats correctly.
- Don't use (for now) the new claimPage function, as it doesn't record the
commits.
- Pass connection parameters as command line options.

TODO:
- Improve performance, possibly using transactions.
- Record performance numbers (bug #46659).
- Remove debugging output.
Bug: 44652

Change-Id: I25ef1f909308676e311180c9cdfceca7f91118fb
Correctly record crashing titles as errors after a number of retries.

Bug #44652

Change-Id: Ib882ba4ece08533e645e3eba6e9a78138f6c2787
Refactor the database schema for performance.

To use only the pages table to determine the next title to be processed,
store the latest claim (hash, timestamp and number of tries) and latest
score for each page.
Then, to get the next title query for the one with the highest score that
doesn't have its latest claim hash equal to the client's hash, except if 10min
have passed (in case of crashing clients).
Also:
- Add a direct reference to the latest result (in addition to the one to the
latest stat) to pages instead of going through claims.
- Remove claims table as it's no longer needed.
- Use (page id, commit hash) as the primary keys for results and stats, as
it allows using INSERT ... ON DUPLICATE KEY to avoid (improbable)
collisions.
- Clean up the code.
- Update the documentation.

Bug: 44652
Change-Id: I5800c99ce2d89c2984ed5376483cdf450f4333fe
Use a local settings file for rt-test server.js, add more options.

The round-trip test server will now read a server.settings.js file for its
options, including MySQL connection settings. The command line options
can still be used and they will override those present in the file.

There are also 3 new configuration parameters:
- 'tries' establishes the number of times a title is tested for a commit hash
before it's considered an error.
- 'fetches' sets the number of fetching errors that will be accepted before
the title is no longer considered for testing.
- 'cutofftime' represents, in seconds, the time to wait for a valid response
from a client.

Bug: 52114
Change-Id: If877269980931157ee2e94996edfa4e473a6f610
Get rid of top regressions / top fixes endpoints in round-trip test server.

Those endpoints are basically useless, as we always use pairwise
comparisons to find regressions. As a bonus, the regressions and fixes
counters now point to the comparisons between the two latest revisions.

Bug: 52235
Change-Id: I2ae6ce9fb089cc45eae6d05641c9b89d0aa04ba6
Provide @lh and latest result links on regressions and topfixes pages

Just some convenience links.
Bug: 52555
Change-Id: Id58f39426d21da7243d0f55b9c3e5cc4c78ef276
Update html5 package from upstream.

Up to date with HEAD aredridel/html5/a0fde78d9b8a00a03f09bade00c215db51545842,
which is slightly ahead of v0.3.14.

Previous version seems to be around v0.3.8,
closer to commit aredridel/html5/b43562a48f63ae72a91694dd77ac5b8d97d624e0.

Adds a dependency on html5-entities.

Bug: 47323
Change-Id: Id3f39d4725a4d4c342368fa2e2047ad454e1e8b5
Tweaks to minimizeInlineTags to work with current HTML libs in Parsoid

* Looks like the orig version was printing paths directly instead
  of collecting nodeName (not sure how that version worked). Fixed
  debugging statements to print node-names for paths.

* The orig version was, in one place, using a HTML node as an object
  property name (instead of using nodeName) which no longer worked
  with current HTML libs. Fixed that code.

  This fix now makes the algo functional again.

  Still not deployable without introducing HTML attribute checks.

Change-Id: I8ccbb66b91e5cebba1c5c7b156e848d99d9d4655
Tweak QuoteTransformer to more closely match PHP doQuotes.

We accumulate a chunks list alternating between quote and non-quote content,
like doQuotes does, and can look at more than one previous token's context
when deciding whether to balance quotes.  We still don't match PHP's
quote parser exactly, but we get closer.

Also duplicate doQuotes algorithm for converting quotes to tags; this
avoids depending on the html tree builder and ensures that our quote
sequences match the PHP parser so that we pass parserTests.

We edit some previously-forked parserTests, now that our output is
closer to the PHP parser's output.  Parsoid still leaves empty open/close
tag pairs where PHP strips them, so we need to fork a few new tests as
well.

Bug: 49926
Change-Id: I7e19565734e7ea1366b95471e46753423e208241
Merge "Update html5 package from upstream."
.jshintignore js/contrib/node_modules

Change-Id: I9a835807f56443f197e0ca1bbddc1ddeb91ed56e
Merge "Tweaks to minimizeInlineTags to work with current HTML libs in Parsoid"
Merge "Tweak QuoteTransformer to more closely match PHP doQuotes."
Bug 50603, 44498: Handle templates straddling cell attributes & content

This patch adds a DOM visitor to fix up content like

{|
|{{nom|Bar}}
|title="foo" {{nom|Bar}}
|}

where {{nom|Bar}} expands to attributes and a table cell, for example

style="foo" class="bar"|Bar

The attributes are re-tokenized and stripped from the table contents. The
template encapsulation metadata is moved to the table cell and updated.

This implementation renders and round-trips the templates documented in
- https://en.wikipedia.org/wiki/Template:Table_cell_templates/doc
- http://en.wikipedia.org/wiki/User:Nicolas1981/Sandbox (from bug 44498)

Nevertheless, there are some limitations:
- We assume that attributes don't contain wiki markup (apart from <nowiki>)
  and end up in text or nowiki nodes.

- Only a single table cell is produced / opened by the template that
  contains the attributes. This limitation could be lifted with more
  aggressive re-parsing if really needed in practice.

- There is only a single transclusion in the table cell content. This
  limitation can be lifted with more advanced data-mw construction.

Bug: 50603
Bug: 44498

Change-Id: Ibbd16952784be6768622ed6df65d3475d2c37e91
Merge "Bug 50603, 44498: Handle templates straddling cell attributes & content"
The autoInsertedEnd flags need to go on the open tag (not the close tag).

Change-Id: I661d45dfc1e01f203ed152268cc6e24ac36d6aaf
Record performance numbers during RT testing

Add a perfstats table to keep different performance metrics for each
page and commit. They are kept as key, value pairs so that it'll be easy to
add new stats in the future. The stats are passed from the client to the
server embedded in the test results, using a new tag.

There's also a very basic interface to view the stats at /perfstats.

Bug: 46659
Change-Id: If73731d53cf360b6a25b3d7bb2bc5c30a2a4e4b2
Merge "The autoInsertedEnd flags need to go on the open tag (not the close tag)."