toke.c: Don’t set PL_expect when emitting USE tokens

This is a follow-up to 52d0e95bf.

It is not necessary to set PL_expect (via TERM or OPERATOR) since
tokenize_use has already done it.  In 52d0e95bf I only changed one of
tokenize_use’s callers, and changed it the wrong way.

No behaviour changes.
Set PL_expect less often when parsing semicolons

As it worked before, the parser (perly.y) would set PL_expect to
XSTATE after encountering a statement-terminating semicolon.

Two functions in op.c--package and utilize--had to set the value to
XSTATE as a result.

Also, in the case of a closing brace, the lexer emits an implicit
semicolon followed by '}' (emitted via force_next).  force_next
records the value of PL_expect and restores it when emitting the
token.  So in this case the value of PL_expect was flipping back and
forth between two values.

Instead of having the parser set it to XSTATE, we can have the lexer
set it to XSTATE by default when emitting an explicit semicolon.  (It
was setting it to XTERM.)  The parser can set it to XTERM in the only
place that matters; viz., the header of a for-loop.  This simplifies
things conceptually, and makes the code a whole line shorter.

(The diff stat shows more savings in line count, but that is because
the version of bison I used to regenerate the tables produces smaller
headers than what was already committed.)
Expand a comment in toke.c (eliminate ‘why?’)
toke.c:S_lop: Don’t set PL_expect if PL_nexttoke

Don’t set PL_expect if we have a ‘next token’, because when it is
popped off the pending token stack PL_expect will be set to the
saved value, clobbering this assigned value.  So the assignment is
unnecessary.
toke.c: Rework LOOPX macro

All its callers have the same two lines preceding it, so fold them
into the macro.  Only assign to PL_expect after the force_word call if
force_word did not find a word.  (If it did find one, then force_next
would have recorded the previous PL_expect value, which will clobber
the one we have just assigned.)
toke.c: Skip PL_expect assignment under KEY_require

Only set PL_expect here when force_word has realised there is no word
there and has not called force_next.  Setting PL_expect after a call
to force_next is useless, as force_next causes its saved value to be
restored when the next token is emitted.

Changing this may seem silly, as an unconditional assignment may even
be faster, but this will allow me to eliminate PL_lex_expect and make
PL_expect handling simpler than before.
toke.c: Correct S_ao description

It applies not just to control flow assignment operators, but to all
‘x=’-style operators.
[perl #80368] Fix implicit assignop in qq"a\U="

The bug report explains it all:
> $ perl -e 'print "a\U="'
> Can't modify constant item in concatenation (.) or string at -e line 1, near "print "a\U=""
> Execution of -e aborted due to compilation errors.
>
> The "a\U=" string constant ought to generate ops corresponding roughly to
> "a".uc("=") (which would then be constant-folded).  However, the "=" is
> being interpreted by the tokeniser as part of the concatenation operator,
> producing ops corresponding to "a".=uc("") (which generates the error).
>
> This happens because the implicit concatenation operator is generated
> in toke.c via the Aop() macro, which allows an addition-type operator
> to be mutated into an assignment operator if it is immediately followed
> by an "=".  It should instead be generated via one of the other macros,
> or possibly a new macro, that doesn't allow for mutation to an assignment
> operator.

This commit does the latter.

> There are multiple sites in toke.c making the same mistake.

The other two instances are harmless, but the next commit will change
them for a different reason (avoiding unnecessary PL_expect assign-
ments with a view to eventually removing PL_lex_expect).
toke.c: Touch PL_expect less for implicit [.,] in quotes

When emitting implicit commas and cats, there is no need to set
PL_expect at the same time, since these code paths have already set
it to the correct value.  Also, the two instances of Aop would check
the current parse position for an ‘=’ to make an assignment operator.
But that could never happen in these two code paths, so the check
was a waste.
toke.c: For plugins, don’t set PL_expect if PL_nexttoke

When a parsing plugin finishes parsing its stuff, the lexer may have
emitted one more token than the construct it was parsing (if the
plugin called parsing API functions like parse_fullstmt).  In such
cases, yyunlex has pushed that token on to the pending token stack
with force_next.

When the lexer is about to emit the plugin’s parsed statement or
expression, if there is a pending token, then it does not need to set
PL_expect, since the previous value will be restored anyway when the
pending token is emitted.

The next commit will disable that save-and-restore mechanism for
PL_expect, so we must not assign to it here.
Stop using the value of PL_expect

The changes in commits leading up to this one avoided unnecessary
PL_expect assignments that would soon be clobbered by this
‘PL_expect = PL_lex_expect’ that restores the previous value.

Hence, we no longer even need to read the value of PL_lex_expect since
PL_expect hasn’t changed.

Just one piece of code (KEY_package) was setting PL_lex_expect
directly instead of having force_next copy it from PL_expect, so this
commit changes it to set PL_expect to the correct value.
toke.c: Consolidate some PL_expect assignments

The previous commit allows these settings of PL_expect to be combined.
We no longer need one before force_next in each instance.
Stop setting PL_lex_expect

As of two commits ago, nothing uses its value any more.
toke.c: Remove PL_lex_expect define

This is no longer used.
parser.h: Comment that lex_expect is unused

There is at least one CPAN module (Data::Alias) that assigns to this.
Removing it won’t shrink the parser struct because of alignment, so
it doesn’t gain us anything.  Just leave it for now.  We can remove
it later if we have to.
Set PL_expect only once after curly subscripts

When curly subscripts are parsed, the lexer (toke.c:yylex) notes that
the value of PL_expect needs to be set to XSTATE (expecting a state-
ment) after the final brace.  When the final brace is encountered,
PL_expect is set to that recorded value.  But then the parser
(perly.y) sets it to XOPERATOR immediately thereafter.

This approach requires a plethora of identical statements in perly.y.
If we just set PL_expect to the right value to begin with, we can
avoid all those assignments.
In perly.y, change PL_parser to parser

All these code snippets are embedded inside a function
(perly.c:yyparse) that puts the current value of PL_parser in a local
variable named parser.  So the two are equivalent, but the latter
only has to access a local variable.

Before:

$ ls -ld perly.o
-rw-r--r--  1 sprout  staff  94748 Aug 22 06:12 perly.o

After:

$ ls -ld perly.o
-rw-r--r--  1 sprout  staff  94340 Aug 22 06:15 perly.o
[Merge] Eliminate PL_lex_expect; fix one bug; other clean-up

PL_expect (PL_parser->expect) is what the lexer uses to keep track of
what type of thing to expect next.  This (partly) determines whether
‘{’ begins a block, or an anonymous hash, or a subscript.

In numerous cases PL_expect was being set to the wrong value.  There
were extra statements to set it back to the right value.  There was
also a mechanism to save the value (force_next/PL_lex_expect) and
restore it later.

If we just set PL_expect to the correct values to begin with, we can
simplify things conceptually and reduce the amount of code.

I fixed bug #80368 in the process, since it got in the way.

I also fixed up some comments in toke.c and changed PL_parser to
parser in perly.y.