Temporarily remove overload.t changes

undo commits done to lib/overload.t on the blead branch
between 2012/03/31 and 2012/06/04 to make rebasing the re_eval
branch easier. These changes will be re-applied at the end of the rebase
Revert two commits to make rebasing easier

These two commits will be added back later after rebasing

    "[perl #108780] Make /foo$qr/ work under ‘no overloading’"

        37c07a4b201c9f799808d346817c4685e3e9002a.

    "regcomp.c: Silence valgrind warning"

        44bed85634c450533d59ae9e371037dde0101d51.
Temporarily revert some lib/overload.t tests

these accumulated from a few recent commits. Removing will make rebasing
easier.
revert a trailing whitespace removal

this is to make rebasing easier. On one of the rebased commits
also removes this whitespace
Revert two commits that fix a VOL declaration.
This has also been done on my davem/re_eval branch,
and its easiest to revert these at the start of my branch

Revert "regcomp.c: Silence compiler warning about longjump"

This reverts commit 24efd69ba77ba76cd714519dccee88f45820d8b4.

Revert "Fix volatile declaration"

This reverts commit 4d6b28934825c9c735195140271a6f93f9c07348.
Revert 4 regex commits to ease rebasing

Revert "Remove some repeated code in pp_regcomp"
This reverts commit 3e1022372a8200bc4c7354e0f588c7f71584a888.

Revert "regcomp.c: Use no_mg for 2nd fetch of pattern"
This reverts commit 3e0b93e82af0f1a033bcdb918b413113f1d61cf0.
`
Revert "PATCH: [perl #101940]: BBC Tk"
This reverts commit 11951bcbfcaf4c260b0da0421e72fc80b4654f17.

Revert "Fix =~ $str_overloaded (5.10 regression)"
This reverts commit 15d9c083b08647e489d279a1059b4f14a3df187b.

These four recent commits on the blead branch overlap with work on the
re_eval branch. To make rebasing re_eval easier, revert them at the
beginning of the re_eval branch. Any remaining value will be re-added
later in the re_eval branch.
correct comment about how strings are tokenised

The stuff about "foo\lbar" being tokenised as a list which you need to
apply join() to, was wrong; the tokeniser outputs the necessary concats
rather than commas.
make re_evals be seen by the toker/parser

This commit is a first step to making the handling of (/(?{...})/ more sane.
But see the big proviso at the end.

Currently a patten like /a(?{...})b/ is uninterpreted by the lexer and
parser, and is instead passed as-is to the regex compiler, which is
responsible for ensuring that the embedded perl code is extracted and
compiled. The only thing the quoted string code in the lexer currently
does is to skip nested matched {}'s, in order to get to end of the code
block and restart looking for interpolated variables, \Q etc.

This commit makes the lexer smarter.

Consider the following pattern:

    /FOO(?{BLOCK})BAR$var/

This is currently tokenised as

    op_match
    (
    op_const["FOO(?{BLOCK})BAR"]
    ,
    $
    "var"
    )

Instead, tokenise it as:

    op_match
    (
    op_const["FOO"]
    ,
    DO
    {
    BLOCK
    ;
    }
    ,
    op_const["(?{BLOCK})"]
    ,
    op_const["BAR"]
    ,
    $
    "var"
    )

This means that BLOCK is itself tokenised and parsed. We also insert
a const into the stream to include the original source text of BLOCK so
that it's available for stringifying qr's etc.

Note that by allowing the lexer/parser direct access to BLOCK, we can now
handle things like
    /(?{"{"})/

This mechanism is similar to the way something like

    "abc $a[foo(q(]))] def"

is currently parsed: the double-quoted string handler in the lexer stops
at $a[, the 'foo(q(]))' is treated as perl code, then at the end control is
passed back to the string handler to handle the ' def'.

This commit includes a new error message:

    Sequence (?{...}) not terminated with ')'

since when control is passed back to the quoted-string handler, it expects
to find the ')' as the next char. This new error mostly replaces the old

    Sequence (?{...}) not terminated or not {}-balanced in regex

Big proviso:

This commit updates toke.c to recognise the embedded code, but doesn't
then do anything with it. The parser will pass both a compiled do block
and a const for each embedded (?{..}), and Perl_pmruntime just throws
away the do block and keeps the constant text instead which is passed to
the regex compiler. So currently each code block gets compiled twice (!)
with two sets of warnings etc. The next stage will be to pass these do
blocks to the regex compiler.

This commit is based on a patch I had originally worked up about 6 years
ago and has been sitting bit-rotting ever since.
update diagnostics message

The description of the "Sequence (?{...}) not terminated with ')'" error
was cut and paste from another error without the text being updated!
disable lexing of (?{}) within \Q, \U etc

when (?{..}) or (??{..}) is enclosed in one of the modifiers like \Q,
don't parse it; treat it as plain characters that are modified by \Q etc.
This means that any code blocks will be interpreted by the regexp compiler
rather than the lexer. This restores behaviour similar to 5.14 and
earlier, except that the (?{}) is *completely* uninterpreted by the lexer,
whereas 5.14 would skip  till matching {}.
fix the descriptions for pregcomp/re_compile

Some while ago most of the body of Perl_pregcomp was extracted out into
a new sub, Perl_re_compile, but the comments describing the function stayed
with pregcomp. Move them down to re_compile, since they mostly apply to
that sub now.
re_eval and closures: add lots of TODO tests

re_evals currently almost always do the wrong thing as regards what
lexical variable they refer to. This commit adds lots of TODO tests that
show what behaviour I think there should be. Note that because hardly any
of these tests pass yet, I haven't been able to verify whether they have
any subtle typos etc.

The basic philosophy behind these tests is:

* literal code is compiled once at compile-time and shares the same
  lexical environment as its surroundings; i.e.

    /A(?{..$x..})B/

  is like

    /A/ && do {..$x..} && /B/

* qr is treated as a closure: compiling once, but capturing its
  environment anew each time it is instantiated; i.e.

    for my $x (...) { push @r, qr/A(?{..$x..}B)/ }

  is like

    for my $x (...) { push @r, sub { /A/ && do {..$x..} && /B/ } }

* run-time code is recompiled each time the regex is compiled; literal
  code in the same expression isn't recompiled; i.e.

    $code = '(?{ BEGIN{$y++} })';
    for (1..3) { /(?{ BEGIN{$x++}})$code/ }
    # x==1, y==3

* an embedded qr is not stringified, so the qr retains its original
  lexical environment; i.e.

  $x = 1;
  { my $x = 2: $r = qr/(??{$x})/ }
  /A$r/; # matches A2, not A1
add Perl_current_re_engine() function

Abstract out into a separate function the task of finding the current
in-scope regex engine ($^H{regex}). Currently this task is only done in
one place each for compile- and run-time, but shortly we'll need it in
other places too.
add Perl_re_op_compile function

Make Perl_re_compile() a thin wrapper around a new function,
Perl_re_op_compile(). This function can take either a string pattern or a
list of ops. Then make pmruntime() pass a list of ops directly to it, rather
concatenating all the consts into a single string and passing the const to
Perl_re_compile(). For now, Perl_re_op_compile just does the same: if its
passed an op tree rather than an SV, then it just concats the consts.

So this is is just the next step towards eventually allowing the regex
engine to use the ops directly.
Mostly complete fix for literal /(?{..})/ blocks

Change the way that code blocks in patterns are parsed and executed,
especially as regards lexical and scoping behaviour.

(Note that this fix only applies to literal code blocks appearing within
patterns: run-time patterns, and literals within qr//, are still done the
old broken way for now).

This change means that for literal /(?{..})/ and /(??{..})/:

* the code block is now fully parsed in the same pass as the surrounding
  code, which means that the compiler no longer just does a simplistic
  count of balancing {} to find the limits of the code block;
  i.e. stuff like /(?{  $x = "{" })/ now works (in the same way
  that subscripts in double quoted strings always have: "$a{'{'}" )

* Error and warning messages will now appear to emanate from the main body
  rather than an re_eval; e.g. the output from

    #!/usr/bin/perl
    /(?{ warn "boo" })/

has changed from

    boo at (re_eval 1) line 1.

to

    boo at /tmp/p line 2.

* scope and closures now behave as you might expect; for example

        for my $x (qw(a b c)) { "" =~ /(?{ print $x })/ }

  now prints "abc" rather than ""

* with recursion, it now finds the lexical within the appropriate depth
  of pad: this code now prints "012" rather than "000":

    sub recurse {
        my ($n) = @_;
        return if $n > 2;
        "" =~ /^(?{print $n})/;
        recurse($n+1);
    }
    recurse(0);

* an earlier fix that stopped 'my' declarations within code blocks causing
  crashes, required the accumulating of two SAVECOMPPADs on the stack for
  each iteration of the code block; this is no longer needed;

* UNITCHECK blocks within literal code blocks are now run as part of the
  main body of code (run-time code blocks still trigger an immediate
  call to the UNITCHECK block though)

This is all achieved by building upon the efforts of the commits which led
up to this; those altered the parser to parse literal code blocks
directly, but up until now those code blocks were discarded by
Perl_pmruntime and the block re-compiled using the original re_eval
mechanism. As of this commit, for the non-qr and non-runtime variants,
those code blocks are no longer thrown away. Instead:

* the LISTOP generated by the parser, which contains all the code
  blocks plus OP_CONSTs that collectively make up the literal pattern,
  is now stored in a new field in PMOPs, called op_code_list. For example
  in /A(?{BLOCK})C/, the listop stored in op_code_list looks like

    LIST
        PUSHMARK
        CONST['A']
        NULL/special (aka a DO block)
            BLOCK
        CONST['(?{BLOCK})']
        CONST['B']

* each of the code blocks has its last op set to null and is individually
  run through the peephole optimiser, so each one becomes a little
  self-contained block of code, rather than a list of blocks that run into
  each other;

* then in re_op_compile(), we concatenate the list of CONSTs to produce a
  string to be compiled, but at the same time we note any DO blocks and
  note the start and end positions of the corresponding CONST['(?{BLOCK})'];

* (if the current regex engine isn't the built-in perl one, then we just
  throw away the code blocks and pass the concatenated string to the engine)

* then during regex compilation, whenever we encounter a '(?{', we see if
  it matches the index of one of the pre-compiled blocks, and if so, we
  store a pointer to that block in an 'l' data slot, and use the end index
  to skip over the text of the code body. Conversely, if the index doesn't
  match, then we know that it's a run-time pattern and (for now), compile
  it in the old way.

* During execution, when an EVAL op is encountered, if data->what is 'l',
  then we just use the pad that was in effect when the pattern was called;
  i.e. we use the current pad slot of the currently executing CV that the
  pattern is embedded within.
make recent re_eval changes compile under -Dmad

Note that this doesn't mean that MAD output is correct; I haven't tested
that. In particular, I have no idea when and how CURMAD() is supposed to
be used.
Ignore code blocks within /[...]/

Now that RE code blocks are being handled by the lexer rather than the RE
compiler, make sure that stuff like

    /[(?{]/

isn't mis-interpreted as the start of a code block.
make qr/(?{})/ behave with closures

With this commit, qr// with a literal (compile-time) code block
will Do the Right Thing as regards closures and the scope of lexical vars;
in particular, the following now creates three regexes that match 1, 2
and 3:

    for my $i (0..2) {
        push @r, qr/^(??{$i})$/;
    }
    "1" =~ $r[1]; # matches

Previously, $i would be evaluated as undef in all 3 patterns.

This is achieved by wrapping the compilation of the pattern within a
new anonymous CV, which is then attached to the pattern. At run-time
pp_qr() clones the CV as well as copying the REGEXP; and when the code
block is executed, it does so using the pad of the cloned CV.
Which makes everything come out all right in the wash.

The CV is stored in a new field of the REGEXP, called qr_anoncv.

Note that run-time qr//s are still not fixed, e.g. qr/$foo(?{...})/;
nor is it yet fixed where the qr// is embedded within another pattern:
continuing with the code example from above,

    my $i = 99;
    "1"   =~ $r[1];    # bare qr:     matches: correct!
    "X99" =~ /X$r[1]/; # embedded qr: matches: whoops, it's still
					       seeing the wrong $i
In Perl_re_op_compile, make a var volatile

This function includes a setjmp to allow for abort and retry if the
pattern getting compiled suddenly becomes UTF8. My recent changes to
it left one var generating the dreaded "warning: variable ‘pat’ might be
clobbered" warning. So declare it VOL to fix this.
unlink re_eval code blocks from op list

In the list of ops generated by something like /abc(?{...})def/,

	const(abc)
	null/special
	    ...
	const(...)
	const(def)

link the list, but skip the DO blocks. This means that for the runtime
case, we no longer need the temporary measure of deleting the DO blocks,
and it will facilitate the next step of handling literal code at runtime,
i.e. /$runtime(?{...})/.