Fix announceLocation in logger

 * oldids weren't being shown because revision hangs off meta.

Change-Id: Ic34db888fec63f1fc33c1b5b121009a8912c9ae6
Add a bin to the package json

 * Changes the package name: mediawiki-parsoid > parsoid

 * Removes the main entry point to the module since what's listed
   doesn't exist and we aren't prepared to require("parsoid")

 * Adds a bin, parse.js, that can be installed globally with npm link.

 * Should we consider npm publishing?

Change-Id: I64af0768f56e807ec438cedc6a4c9d977b71e8af
(Bug 64017) Fixed longstanding tpl-range overlap detection bug

* Consider this following template range overlap scenario:

  tpl-start-1 --- tpl-start-2 --- tpl-end-1 --- tpl-end-2

  where each of the 4 nodes above are sibling elements at some
  dom level of the document. Clearly, TPL-1 and TPL-2 overlap
  and the two ranges should be merged.

  However, the overlap detection code was using a special
  case check of "prev.end === current.start" which only works
  if tpl-end-1 and tpl-start-2 were the same element but fails
  on the DOM shown above.

  In "normal" transclusion scenarios, these kinds of overlaps
  should not be seen. But, fostering of template content can
  generate DOMs like this.

* The following snippet on enwiki reproduces this bug:
------
{{col-begin}}
{{div col}}
{{col-end}}
------

* This patch also adds a missing initialization of a data struct.
  which occasionally caused failures merging ranges and introduced
  the undefined 'length' error as in Bug 64028.

* This fix eliminates template encapsulation failures on
  enwiki:Dirrty, enwiki:Bangerz, enwiki:Back_to_Black and possibly
  other song-related pages (and possibly other pages as well). I'll
  test all the pages separately and update the bug report.

* This patch also adds a "--trace tplwrap" functionality that emits
  start/end nodes for identified tpl ranges and displays whether
  any of them overlap/nest and get merged.

* Added a simplified test case to parserTests.txt (wt2html fails
  without this patch).

Change-Id: I8c3c78f7437f31ace10bbb0ff213157c94488cbe
Some cleanup + trace dom range identification

* Refactored some common code.

* After wasting an hour tracing a non-bug after being confused by
  different tables marked by identical about-ids, fixed trace
  output to prominently emit tag-ids for tables so it is clear
  that different table elements are being examined.

* Added tracing for finding DOM ranges and marked a potential
  bug with a FIXME.

Change-Id: I19ac27f57ccce31580069279df3fa835aa038d73
First simple fix to selser test generation.

* Doesn't add bare text nodes as children of <BODY> -- wraps
  them in <P>.
* Committing this version for now which already fixes a number
  of tests (164 fewer failures).
* Sorry everyone with pending patches -- you will need to rebase.

Change-Id: Ib29b81c45cc76648ca465e9ca00ee65de261d94a
(Bug 64017) Cleanup nested range and cycle detection.

* Always record outermost enclosing range.

* Prevent cycles while recording nesting of ranges.

* Together, this ensures that nesting cycles are short-circuited
  while recording template range nesting graphs and lets us find
  the outermost enclosing range in all scenarios.

* Enables template encapsulation on enwiki:Indology which has
  a missing {{bottom}} transclusion to close a table. Reduced
  test case is:
---------
{{top}}
{{div col|2}}
{{div col end}}
---------

  Before this patch, template encapsulation would fail on this
  snippet.

* Added parser test that captures this behavior. Renamed another
  test to more accurately reflect what is being tested.

* Tested on:
  - enwiki/Mujigen_Hunter_Fandora from 8c22025b
  - enwiki/Dogma_(studio) from bc7b574d

History
-------
It looks like this took several tries before I/we fully understood
the problem. The history of work on this so far indicates all the
bad assumptions I made about templates, what kind of output might
be produced, about ordering of tpls, ids, fostering, nesting, etc.
Many assumptions were implicit assumptions rather than explicit
assumptions.

Ordered chronologically (reverse of git log order):

 ecf0376c: Fix template wrapping to merge overlapping templates
           (old assumption: never considered merging of ranges)
 606c7b57: Fix template merges in the presence of foster-parenting
           (old assumption: never considered fostering)
 1554f4da: Fix template merging code to get rid of weird hash conflicts
           (old assumption: n^2 checks to detect merging + dom nodes
            can be used as hash keys!)
 861d5993: Fixed merging of nested template ranges
           (old assumption: templates don't nest)
 95f50b42: Sort tpl-ranges by aboutId before adding encap. info to DOM
           (old assumption: tpl-ranges are detected in WT source order)
 683a485a: Improved detection of tpl-content fostered out of a table
           (old assumption: something to do with outer/innermost tpls)
 efe2f968: Dont encapsulate tpls that are nested in other tpls
 92ec4761: Reworked detection of nested and overlapping tpl-ranges
           (old assumption: about-ids are assigned in WT source order
            <ref> tags break this assumption)
           First introduction of algo currently in use
 b979895c: Code cleanup + bug fix in detection of template range nesting
 bc7b574d: Handle flipped tpl-ranges in findTopLevelNonOverlappingRanges
           (old assumption: no fostering of partial tpl content)
 8c22025b: Detect cycles in tpl-range nesting and pick a winner
           (old assumption: two tpl ranges cannot be identical)
 605b3a07: Improvement to detection of nested template ranges
 3f8b7b6c: (Bug 64017): Fixed longstanding tpl-range overlap detection bug
           (old assumption: no true overlap. Start-range of one template
           is identical to end-range of overlapping template)
 <THIS>  : (Bug 64017) Cleanup nested range and cycle detection.

           This is a continuation of 605b3a07 and is more robust.
           Now, the (implicit) Nesting graph is constructed after
           breaking a cycle rather than breaking cycles after constructing
           the graph. This matters because the graph only adds one edge
           per node => the cycle effectively leaves it disconnected from
           the rest of the graph.

Change-Id: Iecce8a2e3e07068be5717bfa60c5390a72157611
Fix a number of link-parsing and serialization issues.

Issues fixed:

* `[[Foo|[http://example.com]]]` is a valid link; the link content is the
literal string `[http://example.com]`; that is, it is not autolinked.
Add code to "delinkify" autolinks inside links, being especially careful
with autonumbered links.

* Properly encode `<>` when present as part of a title.  See the `Link
containing "#<" and "#>"` and `Link containing "<#" and ">#"` parser
test cases.  Note that we do not munge anchor strings; HTML5 allows
all non-space characters in ids (but see bug 57252).

* Split anchor from title correctly, even if there are multiple '#'
characters (bug 63430).  Added an additional parser test for this.

* Improve 'valid title' checking in `MWParserEnvironment#isValidLinkTarget`.
Decode %-escapes so that we can reliably detect bad characters, and ignore
any anchor section since the constraints we're testing apply only to title
characters.  Add a test for "% followed by 2 hexadecimal digits" which is
not allowed in a page title.

* Split off an `autourl` production from the general 'url' production
in the grammar, since the exact character classes recognized in
autolinks differ from the more permissive rules for external link
urls.  In particular single quote and open brace are allowed (bug
49688) and there is no `!(space / eolf)` restriction on `[.:,]` in
external link urls (bug 63947). Document where the characters classes
come from on the PHP side for easier auditing.

* Don't parse illegal entities using the `htmlentity` production.  This
improves our handling of raw ampersands in URLs, etc.

* Similarly to the `url` production tweaks above, improve the
`extlink_preprocessor_text` production to match PHP's `EXT_LINK_URL_CLASS`.
In particular we allow single quotes (see the `External link containing a
single quote. (bug 63947)` test case) and the pipe character (see the
`Images with the "|" character in the comment` test case).

* In WTS, escape urls which are percent-encoded but still not safe for
wikitext -- they are safe for HTML5 but contain characters not in
EXT_LINK_URL_CLASS and so are not yet safe for the PHP parser.  Use a
new `Util.entityEncodeAll` helper, since ordinary html entity encoding
is not guaranteed to escape HTML-safe-but-PHP-unsafe characters.  For
most cases this involved adding a call to `escapeExtLinkURL`, but modified
intrawiki links were previously not being escaped at all; add clauses to
mirror those already present for redirects and simple wiki links.

* In WTS, always entity-encode the contents of a `mw:Entity` span.
Previously, we only entity-encoded the contents if they were not HTML-safe,
which isn't an appropriate test for wikitext.

* Add `html/parsoid` sections to all parser tests using autonumbered links
to account for Parsoid's "empty content" representation of autonumbered links.

New blacklist entries:

* html2wt failures for `encoded ampersand` and `encoded equals` where the
encoding is unnecessarily used inside a URL.  We have no way of marking
<span typeof="mw:Entity"> *inside of an href attribute*.  I left these
as failures (inside of rewriting the test) since we could conceivably add
entity spans in some sort of data-parsoid attribute if we really wanted
clean round-trips here.  (This is not a new failure.)

* Same root cause for html2wt failure in `BUG 337: Escaped self-links
should be bold`: use of entities inside a URL.  In this case it's a
wikilink; we could add <span>s here.  (This is not a new failure.)

* New failure for `IPv6 urls`: we don't support the brackets in IPv6
URLs, and neither does the PHP parser (bug 21261).  In fact, this test
is disabled.  We only passed this test previously because we weren't
properly escaping the brackets in the URL, which are not PHP-safe.

* `Bug 2095: link with pipe and three closing brackets, version 2`:
we're properly parsing this one now, we're just adding `<nowiki>` tags
inside the link label.  Discussed this with subbu, it's likely a
WONTFIX, but in theory we could make our `<nowiki>`ification smarter
to handle this case.  We fail the html2html version as well, because
the `<nowiki>`s add `<span>`s.  (This is not a new failure.)

* `Link containing "#<" and "#>" % as a hex sequences`: we fail html2wt
and html2wt because we prefix the anchor with the current page title.
Not a bug, but perhaps should be fixed in the future.

* `Interwiki links that cannot be represented in wiki syntax` fails html2wt
because we convert underscores to spaces.

* `Parsing an URL`: fails html2wt, this test should probably be rewritten.

There are also a few selser failures.  Some of these are legitimate
issues caused by bug 64300: inserting text before an autolink breaks
the link.

I've also cleaned up the whitelist, rewriting tests where necessary to
add `html/parsoid` clauses and document proper Parsoid output.

Bug: 49688
Bug: 63430
Bug: 63947
Change-Id: If579bff310fd08653f55020c70167929423af5be
Don't convert underscores to spaces on interwiki links.

Change-Id: I9eed0ffc911b45fcc35a7d71a2a2f93e575cd10a
WTS: Code cleanup + minor simplification.

* Get rid of chunkCB in serializer state and pass in a cb into all
  serializeNode calls. This keeps code readable and eliminates the
  extra piece of state in 'SerializerState'.

* This also lets us clean up WTS kickoff code.

* Added state.onSOL output to tracing output for every node.

Change-Id: I704221b2e349115c500c231c67f1b276ae47523a
Dont add fostered mark on metas added around fostered content.

* The fostered content marker is only needed on fostered content,
  not on meta wrappers added around fostered content.

  This marker was tripping up DSR by dropping valid DSR information
  on transclusion wrappers that surrounded fostered content and
  was causing template encap failures (because of missing DSR).

  Noticed this on enwiki:Public_holidays_in_Switzerland from
  bug 64017.

* Introduces DSR changes on already blacklisted entries.

Change-Id: I14b3a26099ab1e0d510f2a0d1b40cf30987d6f4d
Logging: Stack traces only for error & fatal events.

* Production logs are getting clogged with cache miss warning
  stack traces.

Change-Id: I205f3461ada998e24661c40f55308d812027632a
Merge "Logging: Stack traces only for error & fatal events."
Merge "WTS: Code cleanup + minor simplification."
Merge "Dont add fostered mark on metas added around fostered content."
Merge "(Bug 64017) Cleanup nested range and cycle detection."
Fix RT crasher caused by pipe characters in link text.

Added parser test to exercise this case, so we won't have to use RT to
catch this in the future.  This test case fails wt2wt, html2html, and
html2wt currently because it adds extra <nowiki> tags to the link content
during WTS (which manifest as <span> tags in the html).  These <nowiki>s
aren't strictly necessary, as discussed for the `BUG 2095` test case in
36053d3b568145d991d5088e1dbe561d60241f02.  So I'm leaving the failures
on the blacklist to indicate that we might fix this in the future.

Change-Id: If047537f106c0a7ae7b54ac69230345187b00ff3
Merge "Fix RT crasher caused by pipe characters in link text."
Minor: Move nested functions to top level.

Change-Id: Ia6acad5b9bd635f8b5de9014b3c24ae06d90a834
Bump the maximum number of API sockets to 50

With HTML parameter expansions we can benefit from more parallelism.

Change-Id: Iafc0522a7123956a15e59f39124301a5e3f0a52f
Inline text_char in urltext production

Change-Id: I12285fab509c1d18e3addc785a7c958f7841e03e