Windows sys.argv again: sys.exit(1) when too complicated, and deliver a looooooong stderr message

I noted last time that my "fix" could not cover all possibilities. After further thought, I decided that the best thing to do in the hard cases is to sys.exit, and give users a clear explanation in stderr so they can re-enter their paths correctly. I may have gotten carried away, but given that many users nowadays are unfamiliar with the command line (even moreso on Windows), I wanted to give them plenty of hand-holding.

I looked up this issue, and it turns out to be a Windows shell problem after all, not Python, which surprised me.
copy original wmllint 1.4 into data/tools

Since esr is overwhelmingly the predominant author of early wmllint, I am importing it in the form of a patch from him, so that git blame will bear some resemblance to reality. Those interested in the full development history should refer back to the 1.4 branch.

git blame (without switches) ascribes the following lines to authors other than esr:

48a82a6f5 data/tools/wmllint-1.4   (Elias Pschernig 2007-05-31 19:06:25 +0000   77) from wesnoth.wmltools import *
4be7e8163 data/tools/wmllint-1.4   (Patrick Parker  2007-09-09 06:12:43 +0000   78) from wesnoth.wmliterator import *
68cb53470 data/tools/wmllint-1.4   (Gunter Labes    2007-07-11 21:09:36 +0000  261)         ("holy=", "arcane="),
68cb53470 data/tools/wmllint-1.4   (Gunter Labes    2007-07-11 21:09:36 +0000  291)     ),
c15a9fc9a data/tools/wmllint-1.4   (Gunter Labes    2007-09-20 16:43:35 +0000  295) 	("Halbardier" , "Halberdier"),
ea3323d98 data/tools/wmllint-1.4   (Patrick Parker  2007-09-06 01:57:09 +0000  948)     global versions
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1071)                 break
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1206)     # Garbage-collect any empty [attack] scopes left behind;
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1207)     # this is likely to happen with female-variant units.
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1208)     nullattack = True
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1209)     while nullattack:
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1210)         nullattack = False
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1211)         for i in range(len(lines)-1):
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1212)             if lines[i].strip() == "[attack]" and lines[i+1].strip() == "[/attack]":
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1213)                 nullattack = True
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1215)         if nullattack:
d84b7f63b data/tools/wmllint-1.4   (Jérémy Rosen    2007-09-08 07:51:22 +0000 1216)             lines = lines[:i] + lines[i+2:]
e542a2ada data/tools/wmllint-1.4   (Gunter Labes    2007-09-20 17:29:49 +0000 1469)         name_pos = wmlfind("name=", WmlIterator(lines, filename))
801aa3003 data/tools/wmllint-1.4   (Jérémy Rosen    2007-10-05 19:59:11 +0000 1487) [attack_anim]
801aa3003 data/tools/wmllint-1.4   (Jérémy Rosen    2007-10-05 19:59:11 +0000 1497) [/attack_anim]\
bc4e1d8f3 data/tools/wmllint-1.4   (Patrick Parker  2007-09-09 23:24:29 +0000 1552)     except IndexError:
bc4e1d8f3 data/tools/wmllint-1.4   (Patrick Parker  2007-09-09 23:24:29 +0000 1553)         has_map_content = False
a106067a8 data/tools/wmllint-1.4   (Mark de Wever   2007-05-27 16:19:33 +0000 1602)         if not map_only:
a106067a8 data/tools/wmllint-1.4   (Mark de Wever   2007-05-27 16:19:33 +0000 1603)             line = mfile.pop(0)
a106067a8 data/tools/wmllint-1.4   (Mark de Wever   2007-05-27 16:19:33 +0000 1604)             if verbose >= 3:
a106067a8 data/tools/wmllint-1.4   (Mark de Wever   2007-05-27 16:19:33 +0000 1605)                 sys.stdout.write(line + terminator)
a106067a8 data/tools/wmllint-1.4   (Mark de Wever   2007-05-27 16:19:33 +0000 1606)             lineno += 1
176b29319 data/tools/wmllint-1.4   (Mark de Wever   2007-10-28 10:53:53 +0000 1725)                 # Strip the starting positions out of the edges
176b29319 data/tools/wmllint-1.4   (Mark de Wever   2007-10-28 10:53:53 +0000 1726)                 outermap(lambda n: re.sub(r"[1-9] ", r"", n), outmap)
bb07ea830 data/tools/wmllint-1.4   (Mark de Wever   2007-10-28 09:03:10 +0000 1728)                 outermap(lambda n: n.replace(r"Gg^Fet", r"Gs^Fp"), outmap)
d725c7cdb data/tools/wmllint-1.4   (Gunter Labes    2007-08-24 04:54:04 +0000 1883) Usage: wmllint [options] [dir]
d725c7cdb data/tools/wmllint-1.4   (Gunter Labes    2007-08-24 04:54:04 +0000 1885)     Takes any number of directories as arguments.  Each directory is converted.
d725c7cdb data/tools/wmllint-1.4   (Gunter Labes    2007-08-24 04:54:04 +0000 1886)     If no directories are specified, acts on the current directory.
d725c7cdb data/tools/wmllint-1.4   (Gunter Labes    2007-08-24 04:54:04 +0000 1896)     -D, --diff                 Display diffs between converted and unconverted files.
ea3323d98 data/tools/wmllint-1.4   (Patrick Parker  2007-09-06 01:57:09 +0000 1903)     global versions
backport: removal of files on Windows before --revert and conversion file rename

Windows won't allow a rename to overwrite an existing file (without Error 183, file already exists).
wmllint, wmlscope and wmlindent: fixed bug caused by Windows' command prompt appending double quotes to arguments

This is a backport of Elvish Hunter's 2013-02-23T13:04:12Z fix to 1.11.
wmllint: update old "portraits/core/" paths to "portraits/"

These portraits were moved prior to 1.1.9. That was before ESR joined Wesnoth development in April 2007, which may explain why wmllint didn't cover this change. Nevertheless, even many 1.2 campaigns still have the old "portraits/core" filepaths.

These old paths also keep post-1.4 wmllint from updating portrait paths to their current location, after they were moved again in 1.5.9.
insert rstrip() to fix wmllint-crashing assumption that unit files are using Unix newlines

In Linux, many 1.2 unit files would crash wmllint, with tracebacks pointing to the "assert male/female_end != -1" line. Male/female_end's value is set to -1, and when it does not meet the condition for converting to i (line index position), the assert statement fails. The "assert male_end" error crashes files with gender=male, or no gender= key (thus defaulting to male). The "assert female_end" error is the female counterpart, and also covers units with both genders.

I found that after commenting out these assert statements, wmllint no longer barfed on those files. Studying the problem for this commit, however, I saw that "endswith()" included a newline. Could it simply be choking on DOS carriage returns? Doing a dryrun in Windows, which defaults to universal newlines support, I did not get the crashes. Change to binary mode, the crashes returned. Insert rstrip() and delete the newlines, and the crashes stop!
Correct "AMLA TOUGH" to "AMLA_TOUGH"

Admittedly, only a couple of campaigns have this error, but it keeps {AMLA_TOUGH 3} from being updated to {AMLA_DEFAULT}.
print second half of "duplicated attack" warning to stderr also

This warning could become confusing if split up by stdout or stderr being redirected.
clean up get_hit_sound messages

The first message has a couple of problems. Technically, get_hit_sound is not a tag, and there is a stray quote mark at the end. Also, i+1 points to the line number of the [/unit] tag, which is not particularly helpful information. This can be changed to point to the line of the get_hit_sound attribute.

For the second message, the %d get_hit_sound is an index position, so +1 for the line number.
upgrade old {UNIT} macro to 1.4's {LOYAL_UNIT}

Of course, during 1.5, this macro was renamed to {*NAMED_*LOYAL_UNIT}, but we will stick to upgrading to "1.4", and worry about changing LOYAL_UNIT to NAMED_LOYAL_UNIT in the current wmllint.

The basic "if '{UNIT '" condition is more efficient than subjecting every line to a complex regex. However, it would be theoretically possible for a matching line to fail the substitution. Thus, I used subn() instead of sub(), and only report an upgrade to stdout if there is at least one substitution. In the hypothetical case that no substitution is carried out, I alert the user so they can look into it.

The regular expression looks intimidating, so here's an explanation:

field 1: unit type - The authors of this era seem to have been pretty good about enclosing fields in parentheses, but this part of the regex accounts for all three possibilities: a) parentheses used to enclose; b) quotes used to enclose; c) no enclosure, thus ending with the next space.
field 2: id - Basically a clone of the first field.
field 3: name - Clone of the first two fields, except for allowing a translability underscore in the last two cases (though early UMC seems to all follow the practice of including the translatability underscore with the rest of the name inside parentheses.)
field 4: side - We can expect this to be a number. Old add-ons should all be using single digits, but I allow more than one digit to match anyway. In theory, there could be cases which would break the regex (e.g., enclosing the number in parentheses, a macro substitution), but since I don't know of any, I'll just call it a day.
field 5: x coordinate - Usually this will be a number, but it could also be a variable or a macro substitution.
field 6: y coordinate - Clone of field 6, except with y/Y substituted for x/X.

(A side note: after testing this commit, I noticed that the introduction to hack_syntax called for "set[ting] modcount to nonzero" when modifying lines, there are "modcount += 1" lines throughout the animation transformations, and hack_syntax ends with "return (lines, modcount)". Looking into it, however: a) my code was working fine without it, with no change detected after I tested inserting "modcount += 1"; b) I never figured out just what use wmllint was making of the modcount, despite all the care to increment it upwards; c) the last suite in hack_syntax doesn't use modcount, either, and it turns out to have been written by ESR himself in February 2008, after all the other code.)
give line number of DEFENSE_ANIM too, when warning that get_hit_sound is also present

Building on our earlier fix pointing this message's line number to the get_hit_sound, it would be even more useful to have the line number of the DEFENSE_ANIM as well. To get this, we change the has_defense_anim variable's value from False to None, and True to i. It is possible that a unit might have two macros for male and female variants, in which case we will assume that the get_hit_sound is most likely associated with the first.
incorporate old image_defensive into DEFENSE_ANIM creation and discard, or warn of its presence

While getting rid of the deprecated get_hit_sound, we can get rid of the even more hoary image_defensive.

We do this with new variables that are initially set to None. While in_unit, we look for the 'image_defensive=' key. If encountered, we record the line's index position (image_defensive) and its value (defend_image).

When we hit [/unit], we look back to see if there is already a DEFENSE_ANIM. If so, and its reaction image matches the value recorded in defend_image, we figure there is no need to preserve the old key, and enter it into a list of image_defensive attributes to be deleted (image_done). If we are creating a DEFENSE_ANIM, we use defend_image for the reaction image instead of doubling up the base image. Once this is done, again there is no point in keeping image_defensive around, and it is entered into image_done for deletion.

If neither of these cases is met, we offer warnings that an outdated key is in use.

When all of the file's lines have been iterated through, we can then remove those image_defensive lines that have been marked as unnecessary.
check for presence of [defend] animation with DEFENSE_ANIM

We look for a [frame] inside a [defend] tag. If there is no image_defensive, we also use any image as our defend_image. We warn if there is already a DEFENSE_ANIM. If there is a get_hit_sound, we go through with the transformation, but alert the porter.
move old top-level files to _main.cfg

We consider a file a main file if it contains either [campaign], [binary_path], or [textdomain]. (Almost all mainfiles have at least one of those.)

Eventually, we check that the campaign directory exists and that the _main.cfg doesn't already exist. If true, we rename.
wmllint: activate wildcard globbing on Windows

The Windows cmd shell does not expand wildcards by default, unlike UNIX shells. This imports glob.glob and runs arguments through it on Windows.

This will be frontported once I'm done on this branch and ready to check out master again.
check that there are actually arguments before running Windows block on arguments

I realized that there was no need to glob Windows arguments if there were no arguments. This meant moving my previous code block above the "if not arguments" statement, which actually creates an argument. And it meant moving Elvish Hunter's code, since the double quote issue will stop my block from working. Once this decision was made, it made sense to put both code blocks under the same "if" conditions, and to check if there were actually any wildcards during EH's block, before running the arguments through glob.
skip further operations if the value is zero-length

This came to my attention because of a Dark Elves scenario with a 'description=' key that was left blank for the value, crashing wmllint with an index error. More broadly, however, the operations in this section are pointless when there is no value.
animation conversion: add in_variation to list of variables defined at start as False

All of these variables are again defined as False when [unit] is in the line, but in_variation was missing from this earlier list. This caused wmllint to crash with an UnboundLocalError on a page of (UtBS) Kaleh-style macros that had no [unit] tag.
fix typo that prevented firststrike from being upconverted

r -> s and no more "Don't know how to convert 'firststrike'" messages.
enable wmllint to convert more old abilities and specials

I noticed that there were some additional weapon specials (marksman) and abilities (nightstalk, steadfast) that also had macros. I also saw examples of ability= keys that had comma-separated multiple values.