toke.c: Fix confused interp-in-format parsing

The lexer keeps track of how many levels of brackets it is currently
in (including the = and . delimiting formats, which count as brackets)
in PL_lex_brackets.

When parsing a format, the lexer sets lex_formbrack to the form’s
bracket number.  Whitespace parsing takes note of this, and avoids
reading beyond the end of the current line inside a format, unless it
is in an inner bracketed construct.  So this:

format =
@
$a
 + $b
.

treats ‘ + $b’ as a literal string, which becomes the second line
of output.

But this:

format =
@
{ $a
   + $b }
.

treats the whole {...} block as the argument for the first format pic-
ture line.

For interpolating (i.e., quoted) constructs, the lexer temporarily
resets the bracket count.  This is so that when it reaches the final
brace in "${...}foo()" it can parse the rest (foo()) as a constant,
rather than a sub call.

When the bracket count was reset, lex_formbrack was left as it was,
resulting in confusion:

# ok
format =
@
${; use
    strict}
.

# also ok
format =
@
"${; {use
      strict} }"
.

# syntax error (or crash)
format =
@
"${; use
     strict }"
.

# crash
format =
@
"${ use
    strict }"
.

When the bracket count is localised and reset at the start of a quoted
construct, we now do the same with lex_formbrack.
Add freed ops to PL_op_(name|desc)

This is useful for debugging, especially with -DT.
Don’t create formats after compilation errors

Otherwise you can end up with a format that has nonsensical op tree,
but it gets attached and you can call it anyway.
toke.c: Add missing debug tokens
toke.c: Remove TOKENTYPE_GVVAL

It has been unused since it was added in bbf60fe6.
Prevent double frees/crashes with format syntax errs

This was brought up in ticket #43425.

The new slab allocator for ops (8be227ab5e) makes a CV responsible for
cleaning up its ops if it is freed prematurely (before the root is
attached).

Certain syntax errors involving formats can cause the parser to free
the CV owning an op when that op is on the PL_nextval stack.

This happens in these cases:

format =
@
use; format
strict
.

format =
@
;use
strict
.

format foo require bar

In the first two cases it is the line containing ‘strict’ that is
being interpreted as a format picture line and being fed through
force_next by S_scan_formline.

Then the error condition kicks in after the force, causing a
LEAVE_SCOPE in the parser (perly.c) which frees the sub owning the
const op for the format picture.  Then a token with a freed op is fed
to the parser.

To make this clearer, when the second case above is parsed, the tokens
produced are as follows:

format =
[;] [formline] "@\n" [,]
; use [<word>]
[;] [formline] <freed>
[;] .

Notice how there is an implicit semicolon before each (implicit)
formline.  Notice also how there is an implicit semicolon before the
final dot.

The <freed> thing represents the "strict\n" constant after it has been
freed.  (-DT displays it as THING(opval=op_freed).)

When the implicit semicolon is emitted before a formline, the next two
tokens (formline and the string constant for this format picture line)
are put on to the PL_nextval stack via force_next.

It is when the implicit semicolon before "strict\n" is emitted that
the parser sees the error (there is only one path through the gram-
mar that uses the USE token, and it must have two WORDs following it;
therefore a semicolon after one WORD is an immediate error), calling
LEAVE_SCOPE, which frees the sub created by ‘use’, which owns the
const op on the PL_nextval stack containing the word "strict" and con-
sequently frees it.

I thought I could fix this by putting an implicit do { ... } around
the argument line.  (This would fix another bug, whereby the argument
line in a format can ‘leak out’ of the formline(...).)  But this does
not solve anything, as we end up with four tokens ( } ; formline
const ) on the PL_nextval stack when we emit the implicit semicolon
after ‘use’, instead of two.

format=
@
;use
strict
.

will turn into

format =
[;] [formline] "@\n" [,]
[do] [{] ; use [<word>] [;] [}]
[;] [formline] "strict\n"/<freed>
[;] .

It is when the lexer reaches "strict" that it will emit the semicolon
after the use.  So we will be in the same situation as before.

So fixing the fact that the argument line can ‘leak out’ of the
formline and start a new statement won’t solve this particu-
lar problem.

I tried eliminating the LEAVE_SCOPE.  (See
<https://rt.perl.org/rt3/Ticket/Display.html?id=43425#txn-273447>
where Dave Mitchell explains that the LEAVE_SCOPE is not strictly nec-
essary, but it is ‘still good to ensure that the savestack gets cor-
rectly popped during error recovery’.)  That does not help, because
the lexer itself does ENTER/LEAVE to make sure form_lex_state and
lex_formbrack get restored properly after the lexer exits the format
(see 583c9d5cccf and 64a408986cf).

So when the final dot is reached, the ‘use’ CV is freed.  Then an op
tree that includes the now-freed "strict\n" const op is passed to
newFORM, which tries to do op_free(block) (as of 3 commits ago; before
that the errors were more catastrophic), and ends up freeing an op
belonging to a freed slab.

Removing LEAVE_SCOPE did actually fix ‘format foo require bar’,
because there is no ENTER/LEAVE involved there, as the = (ENTER) has
not been reached yet.  It was failing because ‘require bar’ would call
force_next for "bar", and then feed a REQUIRE token to the parser,
which would immediately see the error and call LEAVE_SCOPE (free-
ing the format), with the "bar" (belonging to the format’s slab)
still pending.

The final solution I came up with was to reuse an mechanism I came up
with earlier.  Since the savestack may cause ops to outlive their CVs
due to SAVEFREEOP, opslab_force_free (called when an incomplete CV is
freed prematurely) will skip any op with o->op_savestack set.  The
nextval stack can use the same flag.  To make sure nothing goes awry
(we don’t want the same op on the nextval stack and the savestack at
the same time), I added a few assertions.
toke.c: Remove unnecessary assignment

This was made unnecessary in commit 64a408986cf3d, which changed the
enclosing if block’s protasis to PL_expect == XBLOCK (and also made
sure that PL_expect is set to XBLOCK when a format’s special = delim-
iter is expected).
Increase $B::Concise::VERSION to 0.92
B::Concise: Fix -nobanner

It was only working when B::Concise was passed a code ref.
B::Concise: Dump formats upon request
Don’t let format arguments ‘leak out’ of formline

When parsing formats, the lexer invents tokens to feed to the parser.

So when the lexer dissects this:

format =
@<<<< @>>>>
$foo, $bar, $baz
.

The parser actually sees this (the parser knows that = . is like { }):

format =
; formline "@<<<< @>>>>\n", $foo, $bar, $baz;
.

The lexer makes no effort to make sure that the argument line is con-
tained within formline’s arguments.  To make

{ do_stuff; $foo, bar }

work, the lexer supplies a ‘do’ before the block, if it is
inside a format.

This means that

$a, $b; $c, $d

feeds ($a, $b) to formline, wheras

{ $a, $b; $c, $d }

feeds ($c, $d) to formline.  It also has various other
strange effects:

This script prints "# 0" as I would expect:

print "# ";
format =
@
(0 and die)
.
write

This one, locking parentheses, dies because ‘and’ has low precedence:

print "# ";
format =
@
0 and die
.
write

This does not work:

my $day = "Wed";
format =
@<<<<<<<<<<
({qw[ Sun 0 Mon 1 Tue 2 Wed 3 Thu 4 Fri 5 Sat 6 ]}->{$day})
.
write

You have to do this:

my $day = "Wed";
format =
@<<<<<<<<<<
({my %d = qw[ Sun 0 Mon 1 Tue 2 Wed 3 Thu 4 Fri 5 Sat 6 ]; \%d}->{$day})
.
write

which is very strange and shouldn’t even be valid syntax.

This does not work, because ‘no’ is not allowed in an expression:

use strict;
$::foo = "bar"
format =
@<<<<<<<<<<<
no strict; $foo
.
write;

Putting a block around it makes it work.  Putting a semicolon before
‘no’ stop it from being a syntax error, but it silently does the
wrong thing.

I thought I could fix all these by putting an implicit do { ... }
around the argument line and removing the special-casing for an open-
ing brace, allowing anonymous hashrefs to work in formats, such
that this:

format =
@<<<< @>>>>
$foo, $bar, $baz
.

would turn into this:

format =
; formline "@<<<< @>>>>\n", do { $foo, $bar, $baz; };
.

But that will lead to madness like this ‘working’:

format =
@
}+do{
.

It would also stop lexicals declared in one format line from being
visible in another.

So instead this commit starts being honest with the parser.  We still
have some ‘invented’ tokens, to indicate the start and end of a format
line, but now it is the parser itself that understands a sequence of
format lines, instead of being fed generated code.

So the example above is now presented to the parser like this:

format = ; FORMRBRACK
"@<<<< @>>>>\n" FORMLBRACK $foo, $bar, $baz ; FORMRBRACK
; .

Note about the semicolons:  The parser expects to see a semicolon at
the end of each statement.  So the lexer has to supply one before
FORMRBRACK.  The final dot goes through the same code that handles
closing braces, which generates a semicolon for the same reason.  It’s
easier to make the parser expect a semicolon before the final dot than
to change the } code in the lexer.  We use the } code for . because it
handles the internal variables that keep track of how many nested lev-
els there, what kind, etc.

The extra ;FORMRBRACK after the = is there also to keep the lexer sim-
ple (ahem).  When a newline is encountered during ‘normal’ (as opposed
to format picture) parsing inside a format, that’s when the semicolon
and FORMRBRACK are emitted.  (There was already a semicolon there
before this commit.  I have just added FORMRBRACK in the same spot.)
B::Concise: Document formats
[perl #114018] Let eval close over stale vars in active sub

See also commit cae5dbbe30.

These two lines should never produce different values:

    print $x, "\n";
    print eval '$x', "\n";

But they were producing different values if $x happened to have the
tale flag set.  Even if my in false conditional is not supported (this
was the cause of the bug report), it should still work; and it is
not the only way to get a stale lexical in an active sub (just the
easiest way).

As long as the sub containing the eval is active, the eval should be
able to see the same variables, stale or not.

However, this does get a bit tricky in cases like this, which legiti-
mately warn (from t/lib/warnings/pad):

{
    my $x = 1;
    $y = \$x; # force abandonment rather than clear-in-place at scope exit
    sub f2 { eval '$x' }
}
f2();

In this case the f2 sub does not explicitly close over the $x, so by
the time the eval is reached the ‘right’ $x is gone.

It is only in those cases where the sub containing the eval has
the stale variable in its own pad that we can safely ignore the
stale flag.