toke.c: S_scan_heredoc: prune dead code

This incorrect code (using a pointer after finding it to be null)
is the result of the refactoring in 60f40a3895.  It was trying to
account for a string eval with no line break in it.  But that can’t
happen as of 11076590 (if it could it would crash).

So remove it and add an assertion, along with a comment explaining the
assertion.
Avoid uninit warning for qq|${\<<FOO}|

If a here-doc occurs inside a single-line quote-like operator inside
a file (as opposed to an eval), it produces an uninitialized warning.
The goto I added in commit 99bd9d90 wentto the wrong place.
toke.c:S_scan_heredoc: Put stream-based parser in else block

We currently have the code laid out like this:

    if (peek) {
        ... peek inside the parent linestr buffer
    }
    else if (eval) {
        ... grab the heredoc body from linestr ...
    }
    else
        start with an empty string for the heredoc body

    ... parse the body of the heredoc from the input stream ...

The final bit is inside a while loop whose condition is never true
after either of the first two branches of the if/else has executed.
But the code is very hard to read, and it is difficult to fix bugs, as
code cannot be added before the while loop, and the while loop condi-
tion cannot change, without affecting heredocs in string eval.

So put the final parser inside the else.  Future commits will
depend on this.
Finish fixing here-docs in re-evals

This commit fixes here-docs in single-line re-evals in files (as
opposed to evals) and here-docs in single-line quote-like operators
inside re-evals.

In both cases, the here-doc parser has to look into an outer
lexing scope to find the here-doc body.  And in both cases it
was stomping on PL_linestr (the current line buffer) while
PL_sublex_info.re_eval_start was pointing to an offset in that buffer.
(re_eval_start is used to construct the string to include in the
regexp’s stringification once the lexer reaches the end of the
re-eval.)

Fixing this entails moving re_eval_start and re_eval_str to
PL_parser->lex_shared, making the pre-localised values visible.
This is so that the code that peeks into an outer linestr buffer to
steal the here-doc body can set up re_eval_str in the right scope.
(re_eval_str is used to store the re-eval text when the here-
oc parser has no choice but to modify linestr; see also commit
db4442662555874019.)

It also entails making the stream-based parser (i.e., that reads from
an input stream) leave PL_linestr alone, instead of clobbering it and
then reconstructing part of it afterwards.
Fix here-doc body extraction in eval 's//<<END/'

Outside of string eval, this:

s//<<END/e; print "a
END
b\n";

prints this:

a
b

But when we have string eval involved,

eval 's//<<END/e; print "a
END
b\n"';

we get this:

a

b

Amazing!

The buggy code in question goes back to commit 0244c3a403.

Since PL_linestr only contains the contents of the replacement
("<<END"), it peeks into the outer lexing scope’s linestr buffer, mod-
ifying it in place to remove the here-doc body, by copying everything
after the here-doc back to the spot where the body begins.

It was off by one, however, and left an extra line break.

When the code in question is reached, the variables are set as follows:

bufptr = "; print \"a"...  (just after the s///)
s      = "\nb\\n\""        (newline after the heredoc terminator)

The herewas variable already contains everything after the quote-
like operator containing the <<heredoc marker to the end of the line
including the \n ("; print \"a\n").

But then we concatenate everything from s onwards.  So we end up with
the \n before the here-doc body and the \n from after the here-doc
terminator juxtaposed.

So after using s to extract the re-eval string, we increment s so it
points afer the final newline.
lex.t: Mangle obscenity (albeit euphemistic)

It is harder to hack on perl with someone looking over one’s shoulder
when there are comments like this, even when it is euphemistic in its
use of voiced dental stops instead of the voiceless kind.
Make eval "s//<<END/e" slightly faster

The code that peeks into an outer linestr buffer to find the heredoc
body has to modify that buffer and remove the heredoc body from it.

It copies the text after the quote-like operator up to the end of the
line into a new SV, concatenates the text after the heredoc body into
a new SV, and then copies it back to linestr right after the quote-
like operator.

So, in this example:

eval "s//<<END/e; # jiggles\nfoo\nEND\ndie;"

It ends up copying this:

               "; # jiggles\ndie;\n;"

into this at the position shown:

eval "s//<<END/e; # jiggles\nfoo\nEND\ndie;\n;"
                ^

There is no need for two copies.  And there is no need to copy the
rest of the line where the heredoc marker is.
toke.c:S_scan_heredoc: put the croaking code in one spot
toke.c:scan_heredoc: less pointer fiddling; one less SV

The loop for reading lines of input to find the end of a here-doc has
always checked to see whether the cursor (s) was at the end of the
current buffer:

    while (s >= PL_bufend) {	/* multiple line string? */

(Actually, when it was added in perl 3.000, it was in scanstr and
that loop was not specific to here-docs, but also applied to multi-
line strings.)

The code inside the loop ends up fiddling with s by setting it explic-
itly to the end of the buffer or the end of the here-doc marker, minus
one to make sure it does not coincide with the end of the buffer.

This doesn’t make any sense, and it makes the rest of this function
more complicated.

Because the loop used to be outside the else block, it was also
reached for a here-doc inside a string eval, but the code for that
ensured the condition for the while loop was never true.

Since the while loop set s to one less than it needed to be set to,
in order to break out of it, it had to have s++ just after the loop.
That s++ was reached also by the eval code, which, consequently, had
to adjust its value of s.

That adjustment actually took place farther up in the function, where
the herewas SV was assigned to.  (herewas contains the text after the
here-doc marker to the end of the line.)  The beginning of herewas
would point to the last character of the here-doc marker inside an
eval, so that subtracting SvCUR(herewas) from the buffer end would
result in an adjusted pointer.

herewas is currently not actually used, except for the length.  Until
recently, the text inside it would be copied back into PL_linestr to
recreate where the lexer needed to continue (because PL_linestr was
being clobbered).  That no longer happens.

So we can get rid of herewas altogether.  Since it is in an else
block, the stream-based parser does not need to fiddle pointers to
exit the loop.  It can just break explicitly.  So the s++ can also
go, requiring changes (and simplifications) to the eval code.  The
comment about it being a multiline string is irrelevant and can go,
too.  It dates from when that line was actually in scanstr and applied
to quoted strings containing line breaks.
toke.c:scan_heredoc: Remove unnecessary assignment

Updating PL_bufend after lex_next_chunk is not necessary, as
lex_next_chunk itself does it.
toke.c:scan_heredoc: Merge two adjacent #ifdefs
toke.c:scan_heredoc: Remove incorrect part of comment

I missed this in 60f40a3895 when I stopped abusing IVX and NVX.
toke.c:scan_heredoc: Merge similar code

The code for looking in outer lexing scopes was mostly identical to
the code for looking in PL_linestr.
toke.c:scan_heredoc: comments, comments
toke.c: PL_in_eval purge

Many uses of PL_in_eval in toke.c are redundant.

PL_in_eval indicates not that we are parsing a string eval, but that
we are being called from an eval, whether stringy on not.  Even if
PL_in_eval were only for string eval, it would still not indicate that
we are parsing a string eval, because of eval 'require'.

This commit removes redundant uses of it (making things theoretically
slightly faster).
Fix two minor s//.../e parsing bugs

It may be an odd place to allow comments, but s//"" # hello/e has\
always worked, *unless* there happens to be a null before the first #.

scan_subst in toke.c wraps the replacement text in do { ... } when the
/e flag is present.

It was adding a line break before the final } if the replacement text
contained #, because otherwise the } would be commented out.

But to find the # it was using strchr, which stops at the first null.
So eval "s//'\0'#/e" would fail.

It makes little sense to me to check whether the replacement contains
# before adding the line break.  It would be faster just to add the
line break without checking.

But then I discovered this bug:

s//"#" . <<END/e;
foo
END
__END__
Can't find string terminator "END" anywhere before EOF at - line 1.

So now I have two bugs to fix.

The easiest solution seems to be to omit the line break and make the
comment parser skip the } at the end of a s///e replacement.
Break s//3}->{3/e

This should never have worked:

%_=(_,"Just another ");
$_="Perl hacker,\n";
s//_}->{_/e;print
test.pl: Add skip_without_dynamic_extension
utf8cache.t: Skip only the XS-dependent test
[perl #114410] Reset utf8 pos cache on get

If a scalar is gmagical, then the string buffer could change without
the utf8 pos cache being updated.

So it should respond to get-magic, not just set-magic.  Actually add-
ing get-magic to the utf8 magic vtable would cause all scalars with
this magic to be flagged gmagical.  Instead, in magic_get, we can call
magic_setutf8.