Make pos less volatile when UTF8-ness can change

This was brought up in ticket #114690.

pos checks the length of the string and then its UTF8-ness.  But the
UTF8-ness is not updated by length magic.  So it can get confused if
simply stringifying a match var happens to flip the UTF8 flag:

$ perl -le '"\x{100}a" =~ /(..)/; pos($1) = 2; print pos($1); "$1";
print pos($1)'
2
1

$ perl -le '"\x{100}a" =~ /(.)/; pos($1) = 2; print pos($1); "$1"; print
pos($1)'
1
Malformed UTF-8 character (unexpected end of string) in match position
at -e line 1.
0

As pointed out in that ticket, length magic on scalars cannot work
properly with UTF8, so stop using it.
Deprecate mg_length; make it return bytes

mg_length returns the number of bytes if a scalar has length magic,
but the number of characters otherwise.

sv_len_utf8 used to assume that mg_length would return bytes.  The
first mistake was added in commit b76347f2eb, which assumed that
mg_length would return characters.  But it was #ifdeffed out until
commit ffc61ed20e.

Later, commit 5636d518683 met sv_len_utf8’s assumptions by making
mg_length return the length in characters, without accounting for
sv_len, which also used mg_length.

So we ended up with a buggy sv_len that would return a character
count for scalars with get- but not length-magic, and a byte count
otherwise.

In the previous commit, I fixed sv_len not to use mg_length at all.  I
plan shortly to remove any use of mg_length (the one remaining use is
in sv_len_utf8, which is currently not called on magical values).

The reason for removing all calls to mg_length is that the returned
length cannot be converted to characters without access to the PV as
well, which requires get-magic.  So length magic on scalars makes no
sense since the advent of utf8.

This commit restore mg_length to its old behaviour and lists it as
deprecated.  This is mostly cosmetic, as there are no CPAN users.  But
it is in the API, and I don’t know whether we can easily remove it.
sv.c: Document that sv_len sets the UTF8 flag
Make sv_len_utf8 return a character count as documented

Brought up in ticket #114690.

sv_len_utf8 does not make sense.  It assumes that the string is UTF-8.
If it is not, it just does the wrong thing.  For magical variables, it
expects mg_length to return the number of characters, but it would
only sometimes do that until the previous commit, which restored it to
returning  bytes for all scalars.

Since you have to know already that a string is in utf8 before you
can call sv_len_utf8, but sv_len_utf8 might call get-magic which will
change the utf8-ness, it really makes no sense as an API.  Up till
now, it has been consistently buggy with any magic scalars.

So I have changed sv_len_utf8 to do exactly what the documentation
says: return the number of characters.

This also causes an existing buggy code path in sv_len_utf8_nomg to be
reached (SvCUR without checking SvPOK), so this fixes that too.
Stop pos from panicking when overloading changes UTF8ness

This touches on issues raised in tickets #114410 and #114690.

If the UTF8ness of an overloaded string changes with each call, it
will make magic_setpos panic if it tries to stringify the SV multiple
times.  We have to avoid any sv-specific utf8 length functions when
it comes to overloading.  And we should do the same thing for gmagic,
too, to avoid creating caches that will shortly be invalidated.

The test class is very closely based on code written by Nicholas Clark
in a response to #114410.
Stop substr lvalues from being confused by changing UTF8ness
Make magic_setsubstr check UTF8 flag after stringification

By checking it before, it can end up treating a UTF8 string as bytes
when calculating offsets if the UTF8 flag is not turned on until
the target is stringified.  This can happen with overloading and
typeglobs.

This is a regression from 5.14.  5.14 itself was buggy, too, but one
would have to modify the target after creating the substr lvalue but
before assigning to it; and that because of another bug fixed by
83f78d1a27, which was cancelling out this one.

package o {
    use overload '""' => sub { $_[0][0] }
}
my $refee = bless ["\x{100}a"], o::;
my $substr = \substr $refee, -2;
$$substr = "b";
warn $refee;

That prints:

Wide character in warn at - line 7.
Āb at - line 7.

In 5.14 it prints:

b at - line 7.
sv.c:sv_pos_u2b: Don’t cache anything on magical SVs

This should not change the behaviour.  It is just unnecessary to cache
positions on a magical string, as we then just have to invalidate the
cache in mg_get.  It will be more efficient just not to create the
cache to begin with.

This does not allow us to remove the cache invalidation in mg_get,
because the cache can sometimes be created while magic is being called
and SvMAGICAL is off.  It just avoids extra work in those cases that
are detectable.
Make utf8::encode respect magic

It has always ignored set-magic, as far as I can tell.

Since the magic flags patch (4bac9ae47b), it has been ignor-
ing get- magic on magical scalars that were already PVs.
sv_utf8_upgrade_flags_grow begins with an if(!SvPOK(sv)) check, which
used to mean ‘if this scalar is magic or not a string’, but now means
simply ‘if this scalar is not a string’.  SvPOK_nog is the new SvPOK.

Due to the way the flags now work, I had to modify sv_pvutf8n_force
as well, to keep existing tests passing.
Make utf8::decode respect set-magic
Make substr = $utf8 call get-magic once
Only cache utf8 offsets for PVs

References and typeglobs can change their stringification without the
SV itself being assigned to.
pp_length should stringify before checking DO_UTF8

Typeglobs and references can change their UTF8ness upon string-
ification.
sv.c: Don’t cache utf8 length on gmagical SVs

The cache will just be invalidate on the next fetch.  This commit avoids
extra work in those cases that are detectable.  We still have to invali-
date caches in mg_get, because caches can be created while magic is
being called and SvMAGICAL is off.
Update utf8.t tests

Recent commits made utf8::decode and utf8::encode call STORE on tied
variables.  This also causes pos to be reset.  This seems the right
thing to me, as these function actually change the content of the SV,
not just the internal representation.
Remove length magic on scalars

It is not possible to know how to interpret the returned length
without accessing the UTF8 flag, which is not reliable until
the SV has been stringified, which requires get-magic.  So length
magic has not made senses since utf8 support was added.  I have
removed all uses of length magic from the core, so this is now
dead code.
Make 4-arg substr check SvUTF8(target) after stringfying

If it checks the UTF8 flag first, it might be looking at a stale flag,
resulting in malformed UTF8.  Both tests added produced malformed utf8
strings before this commit.

Simply moving this:

	    if (!DO_UTF8(sv))
		sv_utf8_upgrade(sv);

after the stringification is not enough to fix this, as the string
retrieved will be out of date after we do an upgrade.  To avoid
stringifying twice, we use SvPV_force if there is a replacement.  This
means rearranging if() blocks a little.

The use of SvPV_force also means that string overloading is no longer
called twice on the target scalar.  This rearrangement also means
that targets upgraded to utf8 are no longer exempt from the refer-
ence warning.  (Oh, and the test for that warning was not testing any-
thing in its no warnings test, because the target was no longer a ref-
erence; so I corrected the test.)
Make 4-arg substr call FETCH once when upgrading target
sv.c: One less assignment in sv_pvutf8n_force

This function always assigns to *lp twice (though indirectly the first
time).  It only needs to do so once.
Make rvalue substr call overloading once on utf8 str