UESPWiki:Administrator Noticeboard/Archives/Revamping Templates

Revamping Templates

Time to reveal the real reason why I started my recent bout of extension-writing: I think our templates are ugly and broken and I want to fix them.  Just to be clear, I'm not criticizing anyone other than myself with that assessment of our templates: I wrote many of our templates, including the ugliest ones, and I introduced more than a few of the problems.  However, we've all been hampered by limitations of the mediawiki template system -- limitations that can only be overcome by updating the wiki's PHP code.  In other words, I think we need a new extension with multiple template-related tools.

At this point in time, I'm posting this information on the Admin Noticeboard because my target audience is the site's template editors who, given the current complexity of our templates, are admins or at least editors who notice the Admin Noticeboard.  I'm hoping to get feedback on whether some fairly abstract tools (new parser functions and tag functions) seem logical and useful -- and I'm asking for feedback before anyone (other than me) can actually test them or use them.  If you don't understand templates, don't expect to understand the rest of this ;)

First, however, let me provide some perspective with some examples of the problems I think need to be fixed with our templates:
I want to get rid of /Description and /Author subpages.  They cause a multitude of problems, which are only tolerated because there's no other way to have shared information.  Instead, I want to have |description= and |author= fields that are part of our summary templates (e.g.,  ); I want other pages to be able to access those fields -- without having to transclude and process the entire original article.
The new descriptions should be more flexible -- for example, we need find a different way to deal with links on place page descriptions.
Templates need to be more legible, in particular by being less complex.  Ideally non-template-writers should at least be able to get the gist of what's going on in a template; it shouldn't take experienced template-writers twenty minutes to make sense of a template; it shouldn't take two years of UESP experience to learn how to put together a new infobox template.
It needs to be possible to meaningfully test templates when previewing them.

I've put together a set of tools that I think can go a long way towards fixing these issues.  These are tools that will primarily be used within templates -- in many cases, without requiring any changes to our existing articles (except where we want to clean up our articles to take advantage of new features).  I'm also hoping that the tools are semi-intuitive to people who know templates.  Also, to answer the first question that will probably cross your mind reading this list: yes, these features are all feasible.  Most of this is working in the MetaTemplate extension that I'm putting together -- and so far it is a true extension, that does not require any modifications to the core mediawiki code.

The tools:
A #define parser function, that can be used to provide default values for any template variables.
For example, the Faction/Code template would no longer be necessary.  Instead, the Faction template would contain all the code and would start with a set of definitions, such as    .  These are defaults -- so if a specific page specifies    , the value of alt would be "My faction", regardless of altname, faction, etc.  As with many of the following parser functions, this really is more like a parser subroutine: it never returns a value and never displays anything on the page.
Another feature of the #define function is a "case=any" flag.  So     would specify that calling the template with any of imgdesc=, ImgDesc=, Imgdesc=, or IMGDESC= would be equivalent.  No need to constantly repeat   anymore.      combines the two features.

A #preview parser function.  It works exactly the same way as #define, except it only takes effect when you are previewing a template.
With this function, you can test exactly what the template will look like for a given set of input conditions -- but if you then accidentally save the template without cleaning up the #preview lines, you won't mess up pages that use the template.  Or you might intentionally save the #preview lines so that other editors can see a problem situation that needs fixing.

A #inherit parser function that tells a nested template to inherit the listed variables from the template(s) that called it.  This makes nested templates far more powerful and opens up many possible ways to move sections of complex templates into subtemplates.
 For example, on Template:Oblivion Places Summary, the section about welkynd stones and varla stones could be moved into an "Ayleid Ruin Summary Section" template, in which     could appear.  Then, that template could be nested simply as    , instead of    .
So far, the example is just some simplification -- but the idea can then be extended to become truly powerful.  Oblivion Places Summary (or ) could instead call     to insert any type of customized table section for any place type - grottoes, diamond mines, etc.  Place Summary doesn't need to know anything about the requirements of those place types.  Any required parameters only need to be added to the articles describing those places.

A #include parser function that works similarly to the existing Include template.
In the above example,     would be a better way to include the section template, since it would no longer be necessary to create unnecessary summary section templates.

A #save parser function that saves a list of variables to a database table added by this extension.
This is how /Description and /Author pages can be eliminated: the Book Summary template would include a     line.  Or even a     line, in which case game-specific versions of books wouldn't have to repeat all of the information that's already provided on the Lore version of the book.

A #load parser function that gets the information previously saved for a given article using a #save command.
For example, Book Link would start with a     line (where srcpage is my name for the current tt1 parameter, and would be #define'd on the template).  To make the switchover easier, the #load template will also check to see whether a /Description or /Author page exists if there is no #save'd data.

A #nolink parser function.  This one is really a parser function ;) It takes the provided text and strips out all links, leaving just the label.
The raison d'etre for this function is the place description pages.  With this function, we can freely add links to any and all place descriptions -- then use #nolink on  or other pages where the links are overwhelming.  In fact, the links on Oblivion:Places can be turned off by adding a single line at the top of the page (we don't have to modify every place link template on the page), namely    .  To demonstrate how all of these ideas can start to work together, here's one possibility for what the corresponding  template would look like, fully revamped:
 
  } } } 
 
:Most of this post boils down to: is this more legible than the existing template?  It's assuming that mapname and maplink are #save'd as part of the Place Summary/Oblivion Places Summary template -- which means that Place Link should almost never need to call Map Link, and therefore using the Map Link template will no longer crash pages such as . In other words, it's more efficient than our existing template.   It's also using the namespace-related functions added by UespCustomCode (which is why ModName disappears).

A cleanspace tag function, which was already used in the above example.  Within a pair of       tags, any white space in between tags is removed before the page is rendered.
Therefore, #define tags can each be placed on separate lines, #define tags can be separated from #load tags by a blank line, etc. -- all of which make the template more legible.
Another effect is that any categories or breadcrumb trails within the cleanspace tags are disabled on the template page (except in preview mode).  In other words, the cleanspace tags can replace the includeonly tags that are often used on templates, preventing unnecessary redlinks and non-existent categories from being added to the template pages.  However, one critical difference is that the categories and trails are displayed in preview mode -- allowing full debugging of those features, whereas right now mistakes generally don't become visible until after the template has been saved.

A displaycode tag function, whose only purpose is to alter how the template is displayed on the template page.
It's specifically intended for templates such as Hide -- where the template page tells you nothing, so the only way to see what the template does is to edit the page.  Putting       around the code on the hide template would act like a pair of nowiki tags -- but only when looking at the template page.

A cleantable tag function that cleans up any tables found within the       tags.  Specifically, if a row is empty, the row is deleted from the table.  If all of the rows in a table section are deleted, then the section header is also deleted.  (OK, I haven't actually coded up this tag yet, but that's the plan).
This will allow infobox templates to be significantly simplified.  It will no longer be necessary to embed entire sections of the table in #if statements, which in turn means that templates such as     no longer need to be used to build the table, and means that getting the spacing correct within templates will no longer be such an ordeal.

A NESTLEVEL magic word that tells you whether or not the template is being called directly.
This is a somewhat more obscure feature, but since I'd been using nestlevel behind the scenes for some of these features, I figured I'd make it available in case anyone else has uses for it.  When generating the template itself,     returns 0.  When the template is being called directly from an article, NESTLEVEL is 1.  If the template is being called from another template, that is being called from an article, NESTLEVEL is 2, etc.

I know that's a lot of information to digest, so I'm not expecting any immediate feedback.  I wouldn't be surprised if it take a couple of days just to digest the first couple of paragraphs ;) -- I'm asking everyone to start thinking differently about our current templates, to start seeing how they're broken, and then be ready to think about what tools might be necessary to fix them.  I have a couple of years head start, which is how long some of these ideas have been kicking around in my head, albeit with many rounds of revamping along the way.  I'm finally posting some ideas because I think I may at last have come up with some useful ways to actually make some fixes.

However, once people have started to digest the post, there is alot of feedback that would be useful:
First, does any of this seem remotely useful?  Would anyone other than me actually use some of these functions?  Basically, is it worth installing an extension that provides these types of functions? 
Second, do the names make sense to anyone other than me?  Are there better names that could be used for some of the functions?  Are there ways to make the functions more intuitive?  I'd like to get any renaming/major refactoring taken care of early on, before renaming will break any templates that might start to use the functions.
There is one very unorthodox element introduced by this extension: templates using these functions will no longer look the same in preview mode as they do after you save them.  I can see how it might be completely confusing to hit save and see something dramatically different than your last preview.  However, I think it's also necessary for meaningfully debugging templates -- we want accurate previews, but we don't want redlinks and broken categories to actually appear on the templates.  So, is that too confusing or acceptable?
Do you think these functions will be able to do what I'm claiming?  OK, that's not really a fair question.  There are a bunch of details that I haven't covered (to prevent information overload!), which are often important in making the functions really work as expected.  Furthermore, nobody other than me can test them yet.  Nevertheless, if you do think of any snafus, let me know.  Once (if) the extension gets installed, it will be easier to start thinking about this question.
Are there other aspects of our templates that are driving anyone crazy?  Although unless accompanied by concrete ideas of how to fix the problems, it might take me another two years to come up with fixes :P  Nevertheless, it's better to get the ball rolling sooner rather than later.

Take your time digesting, and let me know if/when you have some thoughts! --Nephele  00:24, 20 March 2009 (EDT)

: Well my first thought, just on reading the description for #define was "Wow!" If I had a penny for each time I'd wished I could create variables in templates, I'd have, well... about 63p, but it's still a fantastically useful feature. 
: Question One: can defines refer to other defines? I might just be cross-eyed here but I couldn't see an instance of this in your example. What I mean is, could we change that example to be like this?
  } } }
 
: It would mean complex expressions could be broken down into their component parts to make templates even easier to understand.

: The #inherit function sounds like it could be insanely powerful. There have only been a few cases where I've wanted to do what it allows, but most of those have been recent. The example you described is a good one, is almost exactly what I'd been wanting to do myself, and was probably prompted by the same incident. I can think of a few other places where it could be very useful too.
: Question Two: How deep does the function nest? I don't have a specific case where I'd want this yet, but if A includes B includes C, can C access A's parameters or just B's?

: #include looks good but almost seems like a come-down after the previous two :)
: Question Three: Does it offer the same options as the Include template? That is, can you specify several files to include and it determines which exist, and can you have a prefix? Not an important question as it can all be done with #if statements, but I thought I'd ask.

: #save and #load are the ones I'm going to have to think about most since they probably represent the biggest change. My biggest concern is the impact on the database. A page like  would make one call for each author, one for each description, and it would seem logical to stick the Ids in there too. That would add up to around 750 hits on the database. Now that's obviously one of the worst-case examples, and I know each query would be very quick, but given the popularity of Quest and Place pages that would also use this system it needs thinking about. Does the request create a "batch" of queries and then execute them together. If it could be one query instead of hundreds, that would be great. So that means...
: Question Four: Could we have a few more details about how this works please?
: Having said all that, it's a brilliant feature and I look forward to using it. Anything that gets rid of /Author and /Description pages is good, and I can see the use becoming more widespread as Ids - maybe other data like values and weights - get stored that way too.

: The other additions will be also be massively useful, especially cleanspace and cleantable.
: Question Five: (prompted by the NESTLEVEL magic word) Currently, the wiki software allows a template to transclude itself once (for examples). Does any of this remove that limitation? I've always wanted to be able to use some kind of recursion. The best example would be for the Quest Stages, where formatting is hard-coded on every page. Wouldn't It Be Nice If there was a way of doing something like:
 
:...and have the template calling itself, stripping off the first three params until there were none left? I know there are risks with that, and probably performance implications, but since you're basically re-writing MediaWiki and it's been on my wishlist for a while, I thought I'd ask!

: So in answer to your questions...
: 1) Yes, this seems hugely, massively, incredibly useful. I can't wait to be able to begin playing with the new toys!
: 2) The names all make sense, but I'd change #nolink to #removelinks, #trimlinks or something like that to make it a verb and make it a fraction clearer.
: 3) It's going to seem a little odd at first but it's definitely going to be worth it.
: 4) It sounds like it. This reply, despite its length, is still only a set of first thoughts. I'll try to find some time at the weekend to take a long look at our existing templates and see how this will affect them. I imagine you've already done a lot of that but I want to be clear on how this will change things. The only problem I can see is with the database as described above.
: 5) Templates are never going to be exactly what I'd like. Basically, they're a very complex search and replace routine, and it would be great to have some way of having them more functional - turning them into scripts. Having said that, I never cease to be amazed what can be achieved with them and these changes will make them even more powerful.

: By far the biggest problem with these changes is going to be doing any work on other wikis! I can certainly imagine myself typing "What do you MEAN you don't have the #define extension installed???" at some point in the future.

: The final thought is simply "Thank you!" and "When can we start using them?" –Rpeh  04:49, 20 March 2009 (EDT)

::About this #save: function... If each page gets a saved sortkey, it could allow to remove all the defaultsort instructions; defaultsort would be using the saved sortkey. One thing, though: I suppose that Special:ExpandTemplate (a very useful tool) will be included in the code update, how would this deal with saved values? --Gez 06:23, 20 March 2009 (EDT)

:::Some answers:
:::* Yes, #defines can refer to variables defined by other defines.  My example included one example: place is a defined variable that is then used by srcpage and altname.  The defines just have to be in the correct order -- all of these commands are processed sequentially, so order matters.  Oh, and one other detail about the defines: defines are fully resolved when they are processed.  In other words, if a define includes some complex, database-intensive chunk of code, the processing only needs to be done once, no matter how many times the defined variable is then used.
;::* #inherit looks through the entire stack of calls (if A calls B calls C calls D, D can inherit from A, B, and C) -- but only the calls that led to that template (if A calls B and C, C can inherit from A but not B). It actually crossed my mind that checking the whole stack might be too much sometimes, leading to problems with unwanted side-effects, and I considered adding an option to limit the levels that are checked -- but then I couldn't figure out whether you'd want to be able to limit it to the top-most or bottom-most levels.  So I left limits as something to keep in mind, based upon what examples actually arise.
:::* #include allows multiple files to be checked (and if necessary, more than three files -- as many as you want to list).  However, it doesn't have the "else" option or the "prefix" option of the Template:Include template.  With the ability to do defines, "else" and "prefix" didn't seem necessary any more: you can now do the include once, then freely check the result of the include to decide whether you need to display alternative text, or alter the text.  It isn't really anything new, given that we have a template that can already do those features, but converting it to PHP allows it to be a bit more efficient.
:::* #save only writes to the database once for an article -- even if there are multiple save statements in the article.  All of the information is collected as the templates are processed, but then it is only written once the article has been saved (and once I can therefore get the correct revision ID for that version of the article).  Some other points about save:
:::**  Save statements are always ignored when processing the actual template file (i.e., if   includes a #save statement, it does not create an entry for Template:Book_Summary in the database -- it only creates entries once Book Summary has been transcluded into real articles).
:::** One complication that I've identified and haven't fully resolved yet is what happens if there are multiple sections of an article with separate data to be saved.  For example, if we wanted to save NPC data as part of the NPC Summary, what should be done on pages such as ?  Right now, it will end up just saving data for the final NPC on the page (all the earlier data is rewritten).  Do we want to just have a nosave option in the summary for such cases?  Or do we want the ability to have multiple sets of data for a single article when necessary?  I'm working towards the latter, but all I've really done so far is add the necessary fields to the database table.
:::** The other tricky thing with #save is all of the article management necessary to make it work (page moves, deleted pages, page purges, regenerating pages after templates are updated, etc.).  I've covered some of the bases, but I have a few rounds of debugging to go before feeling confident that all of the bases have been covered.
:::* #load reads from the database once for each article that needs to be read.  At least once the article data is being #saved, it will just be one call; if it needs to then check for a /Description page and an /Author page, that would be two more calls, but that should only be necessary short-term.
:::** The first time a #load call for a given article is encountered, it reads all of the #save'd data for that article and stores the data in memory.  Therefore, even if multiple #load calls refer to the same article, and even if multiple variables need to be read for the same article, it still only needs to read from the database once.  Reading all of the data once means that I only need to do some sanity checks once; the memory overhead is comparable to the current situation.
:::** On pages such as  that will add up to a lot of database calls -- yet that's still significantly fewer than are necessary right now.  Right now, each book listed on Oblivion:Books can potentially require six database calls -- separate calls for /Description and /Author, for each of which it needs to check for the existence of an Oblivion page and a Lore page, then it needs to read the appropriate page.  I'm happy with six times fewer database calls ;)
:::** Trying to combine the queries any further isn't really feasible, as long as I'm working within mediawiki's standard template processing.  I can't easily scan ahead through a document to find all the #load statements (especially since if there are multiple #loads they're likely to all be in templates rather than the main article).  The other option would be to put placeholders in the text as it's being processed, wait until the end of the page to batch-process all of the load statements, then go back and replace all the placeholders.  Mediawiki actually uses placeholders in this way, but I think it would cause more problems than it would solve in this situation.  For example, it would be impossible to then use the description in subsequent #define statements, or in any type of parser function or template processing.
:::** A "quirk" of the #load feature is that, for example,  will look like it's actually transcluding the full content of every book listed on the page.  When you edit the page, every book will be listed under "Templates used on this page"; for each book, the "What links here" list will state that Oblivion:Books is transcluding the page.  However, that's all just because I'm tricking the mediawiki book-keeping.  The reason for the trick is to ensure that Oblivion:Books gets regenerated if the listed books are updated, or even if the Book Summary template gets updated: so that if you change the description of "Arcana Restored", the new description automatically appears everywhere it's supposed to.
:::* Recursive template calls.... It's possible that some of my tinkering would make recursive template calls possible in some situations, although if so it's an unintentional side effect, bordering on a bug.  Truly making recursive calls possible would either require tricking mediawiki's anti-recursion checks or directly hacking the code (and so far all of this has been possible without actually editing the mediawiki code, amazingly enough).  Furthermore, I'm a bit reluctant to do so -- I know that the checks are there because of the potential for DoS-type attacks.  An infinite recursive template call inserted into a commonly-used template would kill the server: the server would try to regenerate every page using the template, and each attempt would tie up the server for 15 minutes until the request finally timed out.  Furthermore, such a problem could easily be caused by a typo from a well-intentioned editor or even admin.  As a result, I'd be very uncomfortable making infinite recursion possible -- maybe making a few more levels possible, but I don't think that would really get us anywhere.
:::** For your specific example, my first thought is that a template similar to  might be a better option -- put each line of the table into a separate template.
:::** Another thought that comes to mind is some type of argument-splitting function.  Something that splits the incoming argument list into pieces, then repeatedly calls a template using those pieces.  Something like     that would call    , then    , then    , etc. until it runs out of arguments.  That's very possible.  I'm already accepting unlimited numbers of arguments with some of these functions (or in other words, any limit on the number of arguments is being imposed elsewhere in the system -- I have no idea whether mediawiki or PHP has a maximum number of arguments).  That seems less susceptible to catastrophic mistakes, as long as the code makes sure that the split number is a positive, non-zero integer.  Is this idea worth pursuing?
:::* Once this extension is stable and once I've made sure it works with MW1.14 or MW1.15, I probably will post it on mediawiki.org as a publicly available extension.  That's one reason I've separated it from UespCustomCode, and it's guided some of my decisions about which function/magic word belongs in which extension.  So maybe other wikis will eventually have these functions!
:::* I'm not sure about the saved sortkey -- a #save only happens once for an article, after all of the templates have been processed; other articles wouldn't want to #load the sortkey, because they would have different names.  (The exception being cases such as books where multiple articles share the same title).  The three ideas I've had about DEFAULTSORT are:
:::** Hack the mediawiki code and hardwire a new, better value for the default sortkey, for example SORTABLECORENAME (see #Code Updates), or any other value that is somewhat closer to what we really want by default on the site.  It's a pretty easy hack (and it's even to a mediawiki file that's likely to permanently include UESP customizations).
:::** Hopefully, just converting all of our templates to use SORTABLECORENAME will fix most of the conflicts -- I'm hoping it provides a valid sort key for nearly every page on the site, and therefore overrides will be much less common.
:::** The least-well-formed idea is to alter the priorities for DEFAULTSORT: make it so that DEFAULTSORT provided in the article itself overrides the values in any templates.  That should be possible with some template rewriting, especially using some of these new functions (inherit + define).  It would also be possible with some code hacks -- although I think the first suggestion would be a better code hack and would then mean that the only time DEFAULTSORT should be necessary is in occasional oddly-named articles.
:::* Good point about ExpandTemplate.  That's an extension that we would like to add to the site.  Unfortunately, it's very actively maintained, which means that it generally requires a very up-to-date mediawiki installation (currently it requires "1.14alpha r42593" -- which I think means the released version of 1.14 is good enough, but I'd have to check SVN to really be sure).  So we haven't been able to install it yet (although I just realized that through SVN I could actually get a less-bleeding-edge version of the extension that might perhaps work on our site even before we upgrade....).  In the meantime, I'll work on the assumption that it will eventually be installed and make sure that all of these features are compatible with ExpandTemplate.  For the most part, it should all be transparent.   But I do need to check your point about #save statements -- I'll need to make sure that ExpandTemplate is prevented from writing #save data (although it would still be able to #load data).
:::* I'll see whether it might be possible for Daveh to get at least a preliminary version of this extension installed this weekend -- assuming that the other extension installations go smoothly enough!  Such an installation would be preliminary -- without any more code tweaks, save/load would be available for testing but not stable enough for use in articles and cleantable wouldn't be available, for example.  However, that would be enough to let other editors start experimenting with the functions, and exploring how some of our templates could perhaps be revised.  Which would need to be done before adding any of this to real articles, anyway, so it can't hurt to get a head start.
:::--Nephele  13:46, 20 March 2009 (EDT)

:::: That all sounds fantastic and I'm really looking forward to using it. I think most further questions are going to have to wait until I can start playing for real, but I do have one more now. I've created two sandboxes, here and here to see how I'd go about redoing the Ingredients Summary (first one - not fully finished yet) and then use save/load to #load information on the Ingredients page rather than hard-code it as at the moment. The question is about the cleantable tag. Will it remove the "1st", "2nd" etc rows? For that matter, am I even remotely close to understanding how these extensions work? It's tricky to use something when you can't see the results, but I think I'm on the right track.
:::: I like the idea of the #splitargs function. It would be useful anywhere we have variable-length lists such as NPCs and locations. We'd probably find lots of uses for it if it was around. If it's not a difficult one to code, I'll say "Yes please" to that one. You're right about resursive templates... it's just that it's such an elegant programming technique it's a shame it's not around! –Rpeh  10:41, 22 March 2009 (EDT)

::::: Whoops - forgot this bit... For Dark Brotherhood Murderer and similar pages, how about putting each one on a subpage and then transcluding for the final page? –Rpeh  12:54, 22 March 2009 (EDT)

\=>
:I would like to join the club of supporters of this great idea. Not that I feel that templates should be readable for any editor by definition, it would limit the freedom of what you can do with it. But it would definitely benefit from improvements like this. 
:Instead I will jump right to the content, and make some comments about the suggested functions.

:cleanspace: Is it possible to forcibly include space characters, abuse html's nbsp characters for example? One problem with the templates has always been readability. You can't format your code like you would with a programming language, being able to include line breaks and indents would be very welcome. Sometimes you need that space character though; not having to close the cleanspace tags and then open them all the time would save alot.
:displaycode: I'm not sure I would welcome this approach for template documentation. See . The seperate doc pages work because they seperate template implementation from usage. You just state how a template should be used, and if a person wants to improve, it has to edit the page anyway.
:NESTLEVEL: In combination with allowing recursion this could simplify alot of templates, like rpeh stated. I love recursion... :)

:The other tags work beautifully for me, though I don't think I will be using them all myself. I'm always a bit wary for odd library functions without some degree of order (like PHP has), but I don't think this is the case (yet) here. --Timenn  &lt;   &gt;   17:53, 22 March 2009 (EDT)