Tell JSHint to ignore dots in regexps

Those are valid patterns when used properly. The resulting warnings hid actual
issues at least in the WikitextSerializer.

Change-Id: I5e6519a7e5e7254a43bd05d22d91b12a98fe3f34
Merge "Tell JSHint to ignore dots in regexps"
Fix <!doctype match regexp

Change-Id: I53fae5573d7bd7db8f748de6c3ae50e485e457c1
[Bug 44447] Tokenizer yields to event-loop after every top-level block.

* Currently, the tokenizer parses the whole page at once queueing
  up a lot of async actions in one shot.  This is not so good for
  a few reasons:
  - Tokenized chunks aren't processed further till the whole page
    is parsed.
  - This can lead to the first template requests issues during
    the tokenization to timeout in some cases since the request
    handler might never get scheduled in time.
  - It can result in template requests going out in big bursts
    rather than being spread out over the page parse.

* This patch fixes the tokenizer to yield control back to the
  event loop after every top-level block.  The tokenizer now handles
  this as follows:
  - It schedules a new tokenization event before yielding.
  - It fixes up tsr offsets (on each restart, the source offset
    is zero which would be incorrect relative to the full source).
    The code for this was extracted out of TokenStreamPatcher,
    extended, and moved to mediawiki.Util.js.
  - It records sol state before yielding and restores it on
    resume (by fixing up the sol production).

* The patch also adds a 'fullParse' mode to the tokenizer where
  it does not yield and tokenizes the entire text synchronously.
  This mode is used by the serializer which uses tokenizer
  instances to parse short substrings and assume the tokenization
  to be synchronous.  This is done via processSync call to the
  tokenizer.

* Replaced __parseArgs patch in the mediawiki.pegTokenizer.js
  with a pegArgs object that could have named properties (cb,
  pegTokenizer, srcOffset) and makes the code clearer to follow.

* No change in parser test results from this patch.

* Also verified output on a few pages via 'node server'

Change-Id: Ic82e3048b6f2c0d8d12f45bdd75ec351f770e3cd
Merge "[Bug 44447] Tokenizer yields to event-loop after every top-level block."
Bug fix in AttributeExpander introduced by git SHA 46119efb

* AttributeExpander stopped cloning tokens always -- it started
  cloning them on demand to eliminate useless work/memory use.

  However, this broke RTing of certain pages in 'roundtrip-test.js'
  (not when run via node server for some reason) because the KV
  obj. of original token was being modified via newAttrs object.

  This is because AttributeTransformManager was reusing the original
  KV object from the token's attributes when its k and v properties
  were strings (the common case).  But, in that scenario, there is
  no ksrc and vsrc to be updated either.  So, the fix is to not
  bother updating ksrc and vsrc in newAttrs[i] in AttributeExpander
  if they dont exist.

* Verified by testing with "node roundtrip-test.js Yelena_Romanova"

* No change in parser test results.

Change-Id: I54c53ae4f380532048a59240a03f9e4a7c867d1b
Fix HTML output in several places in ParserService.

* Emit base href in /_rt/ output of the server in api/.
  - /_rt/* urls had broken image display because of
    missing base href in the document header.

* Fix up missing closing tags in a few places.

* Added linebreaks in the output so the html seen via
  'View Source' is readable.

* Fixed and verified.

Change-Id: I72140a4b4f69f9982fb457768f7d904a2145735e
Fixed computeDSR and findBuilderTags dom pass to start with body

* Starting with document.body eliminates autoInserted* and dsr
  values on html, head, and body nodes.

* Explictly added DSR on body since selective serializer seems to
  require this.

* Changed "ERROR" message about cs/s inconsistency in computeNodeDSR
  to a "WARNING" about the inconsistency so it is less alarming.

* No change in parser tests results.

* Verified changes on a couple of pages with RT testing.

Change-Id: Iec05e0e32663d9f1be997612869104c9121bdbbb
Replace jsdom with domino, removing dependency on the native contextify module.

Domino is also reported to be faster, as it doesn't contain all the sandboxing
required to evaluate <script> tags in document context (which we're not using
anyway).

Use domino for all html5 parsing.

Update DOM version check for domino.

Change-Id: Iebf8fc5a20c3e64b4a0a36380525ca85c00c63e0
Remove html5 dependency (use domino parser/serialization instead).

Use domino to implement Util.serializeNode() and Util.parseHTML().

Change-Id: I4a5611de39172fb92b7feea020d4674438220540
Fix up Jasmine tests; remove HTML5 dependency.

The Jasmine tests currently all fail, apparently due largely to a
change in how <nowiki> is processed -- but at least they run now.
Looks like they've been neglected for a while; probably broken since
at least dfcd5a21d1a312b841df4575a175bb7959cc8cff on Jan 14 2013.

Change-Id: I0383917444d054416bf1aaabf7fae470361133a2
Handle multiply-referenced ref-tags without duplicate ref-content.

* When named-ref-tags are used multiple times in a document, the
  Cite.onReferences function was outputting the ref-content once
  for each use in the references section. While this is "harmless"
  for regular ref-content, when the ref-content has templates
  (ex: {[cite .. }}), that template content is duplicated several
  times -- this also means that template expansions seen in the
  output no longer has unique ids. Occasionally, this trips up
  DOM passes (template encapsulation) which make the reasonable
  assumption that template ids are unique. It introduced an
  infinite loop in the code that detects template range nestings
  and overlaps. This is not triggered always, but only in certain
  scenarios. en:Euphoria (Enrique Iglesias album) was a page that
  sent the parser into an infinite loop.

* Now fixed by outputting ref-content once.

* In addition, added an error-check in the DOM post-pass that might
  get stuck in an infinite loop to output an error message.

* No change in parser test results.

* en:Euphoria (Enrique Iglesias album) now parses and rts without
  getting stuck in an infinite loop.  Possibly other pages as well.

Change-Id: I81a055999eefe3ca36dfb8a9cda3fee538ae6c65
Merge "Handle multiply-referenced ref-tags without duplicate ref-content."
Correctly detect modified tokens (bug from commit 46119efb).

* The wikitext: "[a = {{echo|b}}]" was not being properly parsed
  and consequently RT-ed because AttributeExpander was ignoring
  modified tokens (from expanded attributes).

* Fixed and verified.

* The page en:Gigabarrel from which this test case was extracted
  now RTs correctly.

* No change in parser test results -- worth adding a new test.

Change-Id: I71ecf966e6dad9ed874f2cdfb4a8a5d91ec86d2f
Merge "Fix up Jasmine tests; remove HTML5 dependency."
Merge "Correctly detect modified tokens (bug from commit 46119efb)."
Smart-quote serialized HTML for more compact readable output.

Note that we use standard serialized HTML (with double-quoted attributes)
when doing normalization for running tests.  The serializeNode()
output is only used when sending data to the user; it makes it much
easier to read data-parsoid attributes!

Change-Id: I0d86efcfc927a2dea71964d7ed26dc486efc0033
Merge "Smart-quote serialized HTML for more compact readable output."
Basic implementation of purely DOM-based separator handling

This patch set implements a DOM-based separator handling pipeline, but does
not enable it by default yet.

Each element handler can now define a sepNls object like this:

sepNls: {
    before: function(node) { if(node).... (logic).. return {min: 1}; },
    after: id({min: 1, max: 2}),
    firstChild: id({min: 1}),
    lastchild: id({min: 1})
}

The passed-in 'other' element node allows a customization of the result for
pairs of elements etc.

The code path can be exercised and seems to work ok, but is not yet used.
You can comment out the this.testHandleSeparator call in line 2926 to get some
debug prints.

Also, fix JSHint errors in WikitextSerializer and define proper exports in
defines in preparation of moving away from globals.

No change in parserTests, as the new separator code is not used yet.

Change-Id: Ie0aa7530918da9eb584f2039d49adceeb9f56f5f
Merge "Basic implementation of purely DOM-based separator handling"