dfa: optimize memory allocation

* src/dfa.c (epsclosure): get the value of 'visited' from the argument.
(dfaanalyze): Define and allocate variable 'visited'.
(dfastate): Use not 'insert' but 'merge' to insert positions for
state 0 of DFA.
grep: simplify superset

* src/dfa.h (dfahint): Remove decl.
(dfasuperset): New decl.
* src/dfa.c (dfahint): Remove.
(dfassbuild): Rename from dfasuperset.
(dfasuperset): New function.  It returns the superset of D.
* src/dfasearch.c: Use dfasuperset instead of dfahint, and simplify.
grep: adjust timing back to kwset when dfaisfast is true

* src/dfasearch.c (EGexecute): If DFA fails after kwset succeeds,
the code doesn't return to kwset until it reaches the end of the buffer
or finds a match.  Because of this, although some cases speed up,
others slow down.

Adjust the heuristic for switching to the DFA, so that it
is more likely to switch at the right times.
grep: simplify EGexecute further

* src/dfa.c, src/dfa.h (dfasuperset): Arg is now const pointer.
Now pure.
* src/dfasearch.c (EGexecute): Coalesce some duplicate code.
Don't worry about memrchr returning NULL when that's impossible.
grep: fix the bug in previous patch.

* src/dfasearch.c (EGexecute): Do it.
grep: clarify EGexecute slightly

* src/dfasearch.c (EGexecute): Change if-then-else to !if-else-then.
maint: make ChangeLog generation more robust

* Makefile.am (gen-ChangeLog): Sync changes from GNU coreutils,
to ensure exit status is propagated, and to support an optional
git-log-fix file.
grep: make KWset and DFA agree about invalid sequences in patterns

See: http://bugs.gnu.org/17376
* src/dfa.c (dfambcache): Don't cache invalid sequences, because they can't be
represented by wide characters.
(dfambcache, mbs_to_wchar): Return WEOF for invalid sequences.
(ctok): New global variable.
(parse_bracket_exp, atom, match_anychar, match_mb_charset): Don't allow WEOF.
(lex): Set 'ctok'.
* src/kwsearch.c (Fexecute):
* src/searchutils.c (is_mb_middle): Don't check here.
* tests/invalid-multibyte-infloop: Adjust to fixed behavior.
* tests/prefix-of-multibyte: Add test cases for this bug.
tests: improve coverage for prefix-of-multibyte

* tests/prefix-of-multibyte: Also test the regex version.
grep: simplify and fix problems with KWset-DFA agreement patch

* src/dfa.c (dfambcache, parse_bracket_exp): Simplify.
(mbs_to_wchar, wctok, FETCH_WC, match_anychar, match_mb_charset)
(check_matching_with_multibyte_ops, transit_state_consume_1char)
(transit_state, dfaexec): Use wint_t, not wchar_t, so that
WEOF is treated correctly on platforms where WEOF is not a valid
wchar_t value.
(ctok, lex): Use int, not unsigned int, for characters,
so that EOF is treated more naturally.
(parse_bracket_exp): Use NOTCHAR to mark uninitialized char, since
FETCH_WC can now set the char to EOF.
(lex): Remove unnecessary test for EOF.
(parse_bracket_exp, atom): Swap then and else parts, to put
the small one first; this is more readable here.
* src/searchutils.c (is_mb_middle): Simplify.
dfa: minor simplification

* src/dfa.c (parse_bracket_exp): Use enum, not macro, and move var
to just the scope it's needed.
grep: fix encoding-error incompatibilities among regex, DFA, KWset

This follows up to http://bugs.gnu.org/17376 and fixes a different
set of incompatibilities, namely between the regex matcher and the
other matchers, when the pattern contains encoding errors.
The GNU regex matcher is not consistent in this area: sometimes
an encoding error matches only itself, and sometimes it
matches part of a multibyte character.  There is no documentation
for grep's behavior in this area and users don't seem to care,
and it's simpler to defer to the regex matcher for problematic
cases like these.
* NEWS: Document this.
* src/dfa.c (ctok): Remove.  All uses removed.
(parse_bracket_exp, atom): Use BACKREF if a pattern contains
an encoding error, so that the matcher will revert to regex.
* src/dfasearch.c, src/grep.c, src/pcresearch.c, src/searchutils.c:
Don't include dfa.h, since search.h now does that for us.
* src/dfasearch.c (EGexecute):
* src/kwsearch.c (Fexecute): In a UTF-8 locale, there's no need to
worry about matching part of a multibyte character.
* src/grep.c (contains_encoding_error): New static function.
(main): Use it, so that grep -F is consistent with plain fgrep
when the pattern contains an encoding error.
* src/search.h: Include dfa.h, so that kwsearch.c can call using_utf8.
* src/searchutils.c (is_mb_middle): Remove UTF-8-specific code.
Callers now ensure that we are in a non-UTF-8 locale.
The code was clearly wrong, anyway.
* tests/fgrep-infloop, tests/invalid-multibyte-infloop:
* tests/prefix-of-multibyte:
Do not require that grep have a particular behavor for this test.
It's OK to match (exit status 0), not match (exit status 1), or
report an error (exit status 2), since the pattern contains an
encoding error and grep's behavior is not specified for such
patterns.  Test only that KWset, DFA, and regex agree.
* tests/prefix-of-multibyte: Add tests for ABCABC and __..._ABCABC___.
grep: improve internal API for multibyte boundary

* src/search.h, src/searchutils.c (mb_goback): Rename from
is_mb_middle.  Omit last arg.  Return number of bytes to go back,
not just a boolean.  All uses changed.
* src/dfasearch.c (EGexecute):
* src/kwsearch.c (Fexecute): Adjust to API change.
* src/kwsearch.c (Fexecute): Eliminate common subexpression.
grep: fix -w match next to a multibyte letter

* NEWS: Document this.
* src/dfasearch.c, src/kwsearch.c (WCHAR): Remove.
(wordchar): New static function.
* src/dfasearch.c (EGexecute):
* src/kwsearch.c (Fexecute): Use the new functions, so that the
code works correctly if a multibyte character adjacent to the
match has two or more bytes.
* src/search.h, src/searchutils.c (mb_prev_wc, mb_next_wc):
New functions.
* tests/word-delim-multibyte: Add a test for grep -w (which now
passes), and a test for \> (which still fails).  The \< test also
still fails.
dfa: speed up 'dfaisfast'

* src/dfa.c (struct dfa): New member 'has_backref'.
(addtok_mb): Set it.
(dfaisfast): Use it.
dfa: minor performance improvement for previous change

* src/dfa.c (struct dfa): New member 'fast'.  Remove 'has_backref'.
All uses changed.
dfa: clarify use of "if"

The phrase "Y is true if X" is logically equivalent to "X implies Y",
but often "X if and only if Y" was intended.
* src/dfa.c, src/dfa.h: Reword to avoid the incorrect use of "if".
doc: mention performance changes

* NEWS: Discuss recent performance improvements and downgrades.
grep: improve performance of -v when combined with -L, -l or -q

Problem reported by Jörn Hees in: http://bugs.gnu.org/17427
* src/grep.c (grepbuf, grep): When -v is combined with -L, -l, or -q,
don't read data unnecessarily after a non-match is found.
grep: simplify and clarify invert-related code

* src/grep.c (out_invert, prtext): Use bool for booleans.
(prline): Remove unnecessary '!!' on a value that is always 0 or 1.
(prtext): Remove last arg NLINESP; use !out_invert instead.  All uses
changed.  Move decls to nearer uses, since we can assume C99 here.
Update 'outleft' and 'after_last_match' here; it's simpler.
(grepbuf): Compute return value by subtracting new from old 'outleft',
rather than by keeping a separate running total.  Avoid code duplication
by arranging for prtext to be called from one place, not three.