code to change Windows-style backslashes in file paths to frontslashes

The task is to replace Windows backslashes in paths, without indiscriminately replacing backslashes in legitimate use as escapes (or bridge terrain).

Breaking this down:

* "no-syntax-rewrite": I don't think this is really necessary, but I will follow the practice of the hack_syntax section.
* if lines[i].lstrip().startswith("#"): Excludes lines that are only comments or defines.
* precomment: Originally, I simply excluded "#" during the while statement, but I realized that this could wrongly mistake a Pango color code (or old-style markup for green) as a comment. I now look for whitespace before "#", and rewrote this section to operate on precomment rather than the whole line.
* comment: Simply going with the second field would exclude the separator itself, so I use len.
* if '\\': Technically, this code worked going straight to the while statement, but running every line through the complicated regex made it sluggish. Faster to make sure the line meets a simple filter first.
* while: It is possible, though very uncommon, that a line could contain more than one file path. Looking at the regex test: a) Match the backslash itself, then be on the lookout for a character used to set apart a file path from other text. By excluding these, we ensure that there is an unbroken chain from the backslash to the file extension.  b) Then we have a list of file extensions, bracketed on the left by a period and the right by \b, to make sure they do not coincidentally match a string. These are the file types that might be referenced by a value (for instance, translation files are not referred to directly, so their extensions are not included). As a practical matter, EVERY instance in the wild I know of involves png, not counting one commented-out path in an ancient campaign.  c) File extensions can include capitals, particularly on Windows, where the effects of DOS unicase linger. So we make the search case-insensitive.
* regex object: This splits the line into groups. The differences from the regex used in the while statement: a) We also look for a non-pathbreaking string to the left of the backslash as well as the right. This means that group(1) will match the entire file path except the extension. b) The \b boundary has been made a non-consuming look-ahead assertion, to simplify future references to the regex object and its groups.
* precomment: Here, we simply reconstruct precomment in relation to the regex object, except all backslashes in group(1) are replaced by frontslashes.
* print: Besides reporting the substitution to stdout, I include a plea for cross-platform compatibility.
* lines[i]: Now it's time to rebuild the whole line.
regex sub to remove 'userdata/' from paths

The misguided authors who put userdata/ in their paths cause problems not just for non-Windows users, but fellow Windows users who chose not to put userdata in the install directory. This error can be removed by an approach similar to that just used to purge backslashes:

* if 'userdata': A basic filter to cut down the number of lines being run through complicated regex speeds up performance.
* while: It is possible, though rare, for a line to contain more than one path with userdata. Points about the regex: a) We continue to use precomment, though it would be well to correct commented-out old paths also, lest they mislead any more UMC writers.  b) In case you're wondering why I made one 'data/' string optional, there's a set of add-ons in 1.4 that use "userdata/campaigns" instead of "userdata/data/campaigns".  c) The '[ac]' at the end is something of an artifact of the time before I excluded comments, but it provides another safety measure insuring that the string is actually a value.
* regex object: This splits precomment into groups. Notes: a) Some authors begin with an unnecessary "../", might as well get rid of it as well. (As far as I can tell, this prefix has no effect anywhere I've seen it used, but I'd want to be positive that it ALWAYS does nothing before having wmllint replace it everywhere.) b) The first two groups have been made non-capturing; we will not need to refer to them.  c) For reporting to stdout, group(1) is extended to the next '/', though this part of the match is optional, to insure that there's no way to get trapped in a while loop.
* precomment: Here, we reconstruct precomment based on the regex object, except we simply drop what's outside group(1).
* print: In case designers don't get the point from seeing the elimination logged in stdout, I include an all-caps admonition against "userdata/". This is a really irritating bug.
* This code was inserted before the reconstruction of lines[i] from precomment and comment.
update 1.4's {LOYAL_UNIT} macro to {NAMED_LOYAL_UNIT}, using a regex to distinguish it from the current {LOYAL_UNIT}

When 1.4's {LOYAL_UNIT} changed to {NAMED_LOYAL_UNIT}, the old name went to a new macro. However, the current {LOYAL_UNIT} ends after the x,y coordinates, so if a regular expression detects two more blocks of text afterwards, it must be the old macro.

Step by step:

* This is going to be the first of several operations involving lines with macros, so we will start with an if filter looking for lines with an opening brace, and indent the rest of the code inside it.
* Then we look for a regex match. The first field of the macro is the side number, which should be a number. Then we have the unit type field, for which we have three rules, depending on whether the author chose to use parentheses, quotes, or a bare entry. Then we have the x,y fields, which will normally be numbers, but which could also be a variable. Different variable names could be used, but we will assume that there will be a $ sign, and an x (or y) somewhere. Then we get to the key. After the coordinates, there is another string that doesn't include a space or a closing brace, followed by at least one more string without a space or closing brace. For current purposes, we only need to confirm that there are at least two more blocks of text before the right brace, and we needn't concern ourselves if they fit a format.
* If we have a match, the change is a simple replacement.
* Log the change to stdout. I'm a little worried about a newbie seeing the change in the log and getting the idea that ALL instances of {LOYAL_UNIT} should be replaced, so I added a little message that this is the 1.4 version of the macro.
auto-recognize the people in the {NAMED_*UNIT} macros

The heart of this code is a regex object, named, which will be used to split out the group containing the macro's character id field. Going over the regular expression:

* We don't want to process any commented-out macros.
* We handle all the NAMED_*UNIT macros by allowing any string of caps and underscores to match. This could also match UMC macros, like "Attack of the Western Cavalry"'s NAMED_SHIP_UNIT. (In theory, there could be false positives from UMC macros that followed the mainline naming convention but not the format. However, such macros would most likely fail the regex, and even if they didn't, false entries in "present" are generally not a major problem.)
* The first macro field, for the side, should be a number.
* For the unit type field, three rules, depending on whether the author uses parentheses, quotes, or a bare entry. At least one campaign uses these macros to recall units in subsequent scenarios with a blank unit type field, so I do not require there to be something inside the parentheses or quotes.
* Then we have the x,y fields, which will normally be numbers, but which could also be a variable. Different variable names could be used, but we will assume that there will be a $ sign, and an x (or y) somewhere.
* Now we come to the id field, the one we're interested in. As with the unit type field, there is a threefold rule. The only difference is that while there are authors who have left this field blank, there's no point in auto-recognizing those cases, so I do require parentheses and quotes to contain something.
* I also check that there is one more field in the macro. Originally, I also checked for a closing brace, but I became aware that some instances of NAMED_UNIT have WML that continues onto another line.

If there is a match, we strip group(4) of parentheses, quotes, and any waste spaces this leaves, and append it to the list of the present.

I also tweak the wording in the intro about the "wmllint: recognize" magic comment to reflect that wmllint is now recognizing ids in some macros.
add conversion of "~/campaigns" to "~add-ons"

grep turns up several old add-ons that use this for map_data.
convert old UMC paths from data/campaigns/ to data/add-ons/ in a mainline-safe way

My first step is to draw up a list of mainline campaigns to compare path directories against.

As for the conversion itself, I want wmllint to be able to accurately handle a possible case where a line contains both a UMC and a mainline path - even though I don't know of any actual example in the wild. So:

cleanup_main: This will be used to convert mainline paths back to normal, as we shall see.
while: As long as there is any instance of "data/campaigns/<dir>", we will keep replacing.
  dc : This regex object uses the same regex as the while statement, except that the campaign's directory name has been separated out as a group and extended. (It would get the job done with just '\w+', since all mainline campaigns use only letters and underscores, but I've added characters that are used in some UMC names so that the full directory name gets printed to stdout.)
  if: group(1) is checked against the list of mainline campaigns. If it's not there, the line is rebuilt substituting 'data/add-ons/' where 'data/campaigns/' was, and a report printed to stdout.
  else: The alternative means that it is a mainline path. However, we need the 'while' test to go forward, so in this case, a meaningless string is substituted for 'data/campaigns/'. The line is flagged by setting cleanup_main to True.
if cleanup_main: Now we can change mainline paths back to "data/campaigns", a simple replace operation. This means that mainline campaign files will not be touched, since the net effect is to leave the line unchanged.
remove '~' in [textdomain] and [binary_path] paths

Although the original purpose of the in_textdomain and in_binary_path code, an aborted effort to update their paths to "data/add-ons/", has been superseded by code that updates those paths on all lines, it can still be put to use.

Our first step is to move that section below the code that updated UMC paths, so our regexes won't have to deal with "@campaigns" and "data/campaigns" strings. Then we delete the 'if 0:' line that was neutralizing this section, as well as the obsolete path-changing code. The rest is de-indented one level.

Then we look for the use of "~" for userdata, which does not work for textdomains and binary paths.

Our regex object, 'tilde', is constituted thusly: (1) We make sure that the line starts with the "path" key. Any line we're interested in ought to start with this, and this will also keep this code from going wild on the campaign includes, if an author forgot a closing tag (no reset to False). (x) There shouldn't be any whitespace around the = sign, but we'll be kind. (2) On the value side, there shouldn't be anything before the tilde except perhaps a quote. Rather than underestimate the ingenuity of authors in coming up with weird code, however, I allow anything except a comment to match for a few characters. But if we haven't hit the '~' after five characters, I figure something's wrong, and bail. (3) Then we come to the tilde. Normally, it would be adjoining "add-ons/", but some authors interpolate a slash, or 'data/' (here represented as an optional string).

If we match, we rebuild the line, except 'data/add-ons' is substituted for group(3), and we log to stdout.
check for textdomain paths that don't include /translations, and binary paths referring to 'external' data

The in_textdomain and in_binary_path code can also be used to check path directories.

First, the convention that [textdomain] uses "/translations" is strongly established, and I can't think of a legitimate reason for an add-on not to be following it. So if it doesn't, let's print an alert to stdout.

As for binary paths, there is the continuing use of special directories to hold menu images. Although these are non-standard, the names that took hold are "/public" and variations of "/external*". So as a crude test, we can look for those strings, and print an error. (A more sophisticated test would look for binary paths outside the campaign #ifdef.) It does not catch the worst case of all - when all binaries are outside the campaign define, not just a set-aside directory.
insert lstrip() into instances of 'if lines[i].startswith("#"):' in hack_syntax

Surely the intent here would also cover indented comments.
drop the ".mask extension" fatal error and soften the "error" message

As far as I can tell, Wesnoth interprets files based on whether they're called by a map_data= or a mask= key, not its file extension or 'usage=' attribute. So I consider it too drastic to turn this into a wmllint "fatal error".

Thus, I comment out the sys.exit.

For those who want their file to have the proper ending and attribute, I leave a softened version of the introduction's explanation and stdout message.
fix wmllint-crashing map_data= bug (only known to affect Invasion of Arendia)