Wikipedia talk:Merge some redundant lists to categories

message: This page is for comment on the project to merge lists to categories. Go to the project page to add a list to be merged. Philwelch 23:32, 24 Mar 2005 (UTC)
replies: text: Alright, that's two *technical* reasons for retaining lists that could otherwise be categories, even though we have good *conceptual* reasons for switching to categories. How about retaining lists *in the main article*, but keeping them synchronized with the category? Two redundant but identical sets of information are better than two redundant but different sets of information. Philwelch 01:33, 26 Mar 2005 (UTC)

text: No one is discussing merging all lists to all categories. Please read and understand the proposal before you comment on it. The fact is, there are presently actual lists that exist on Wikipedia that are redundant with actual categories, and those particular lists should be merged. Philwelch 18:39, 29 Mar 2005 (UTC)

message: :Myth or fact?
::See the link to comments by Jamesday below. -- Beland 03:23, 31 Mar 2005 (UTC)
thread_title: Categories produce relatively heavy server load

message: :Any proposed solutions?
::Yes - file a bug report and get it fixed. -- Beland 03:23, 31 Mar 2005 (UTC)
thread_title: Watchlists and the History tab cannot track changes in a specified category's articles and subcategory lists

message: :Please elaborate
::You'll have to ask [[[User:John Gohde]] for examples. --  Beland 03:23, 31 Mar 2005 (UTC)
thread_title: Categories appear broken on mirror sites

message: :Any proposed solutions?
::1.) Continue to use lists. 2.) Allow for "hidden" categories.  More discussion below. -- Beland 03:23, 31 Mar 2005 (UTC)
thread_title: If an article is placed in too many categories, there will be too many category links at the bottom of the page

message: :That is an argument for retaining some lists as lists. No one ever suggested merging all lists to categories--that's why I moved this page to "Merge some redundant lists to categories". Philwelch 20:29, 30 Mar 2005 (UTC)
replies: text: Disagree - if they are redundant, they should be merged, otherwise why would there be both? Please show examples. --Yurik 22:50, 30 Mar 2005 (UTC)
replies: text: Well, in these cases, the redundant categories should be converted into lists.  These types of categories are currently being deleted by WP:CFD, so there are probably few, if any, example of categories of this type.  List of famous left-handed people would be an example of a list of this type. -- Beland 03:23, 31 Mar 2005 (UTC)
replies: text: Yurik, you just said you disagreed with me and then restated the point I'm trying to make here. Philwelch 03:29, 31 Mar 2005 (UTC)
replies: text: Philwelch, sorry, misread the argument. Has anyone considered having a "Meta box" next to the main article edit box, so that all categories and interwikies would go there instead of cluttering up the article text? It sometimes gets very disorienting, especially for new authors. --Yurik 08:06, 31 Mar 2005 (UTC)
thread_title: Some aspects that may merit mention in a list of some sort may nonetheless be so tangentially connected to the article that a category link at the bottom of the page is a liability

message: :Agree. Need a feature request (i.e. as part of "move" tab). Maybe limit for the new users - on click show move category request page. --Yurik 22:50, 30 Mar 2005 (UTC)
replies: text: Bots are currently being used as an interim measure. -- Beland 03:23, 31 Mar 2005 (UTC)
thread_title: Categories are harder to move

message: :Please elaborate
::Well, the general style for categories is that if there are too many articles (over 50? 100? 200?), you split the category up into subcategories, to make things easier to find.  This means that the original membership is spread over multiple subcategory pages.  With a list, you would simply divide into sections, so you could still see all the articles in a unified listing, but things would also be easy to find.  You could fix this if you engineered non-alphabetical sorting for categories (and convinced people to use it). -- Beland 03:30, 31 Mar 2005 (UTC)
thread_title: It's nicer to view a long list on a single page with subsections than a category which is split into subcategories on multiple pages

message: :Please elaborate. Isn't search disabled? Google & Yahoo are currently default
::True, the site's search engine has larger problems.  I guess Google does see categories.  I list this here because Jamesday mentioned it in his comments (linked below). -- Beland 03:23, 31 Mar 2005 (UTC)
thread_title: The site search engine doesn't work well with categories

message: * The pointer from a category to a member article is automatically synchronized with the pointer from the article to the category.
* As articles are renamed, merged, and redirects created, the category membership list is automatically updated.
* Complicates vandalism, as only one item in category would get vandalized
:Example - Recent deaths can get vandalized, by adding names to it. Each change would have to get researched by the maintainer. When categories are used, the maintainer of the personal page would see the change, and would likely know more about the subject matter.
:: I'm not sure it makes vandalism harder, exactly...with a list, you can watch the list for vandalism.  With categories, you have to watch the article to catch vandalism.  It's currently not possible to watch all possible articles that might in the future be added to a category, and it's not possible to view the history of changes to the category's membership lists.  With a list, you can watch both the list and all the articles in it.  You can also use "Related changes" to watch all the articles in a list without adding them to your personal watchlist.  But categories could be fixed so that "Related changes" and watchlists and histories work, at which point they would be more clearly superior to lists for publicizing vandalism to more editors. -- Beland 03:35, 31 Mar 2005 (UTC)
replies: text:  With smaller category, individual observer can have expertise over an entire topic with all entries, and will easily spot mis-listed item. With large lists, such all-encompassing knowledge is less likely, thus more items might get in without being thoroughly checked. With categories, the article itself becomes the point of monitoring, and original article's author might know better which category it should be subscribed to.  If we get category monitoring, the whole point is mute, as both category-side AND article-side monitoring becomes possible. --Yurik 08:20, 31 Mar 2005 (UTC)
thread_title: Arguments for converting lists into categories

message: From Jamesday's comments
here and
here, it sounds like the performance problems are caused simply because category page views are not cached.  This means that if 1,000 people view a category, then 1,000 database lookups are needed.  We would very much prefer that, say, 990 of those people view a cached version, and do only 10 database lookups.  Am I wrong in thinking that constructing a category page from scratch isn't significantly more expensive than constructing a list page from scratch?  Either way, it needs to be determined whether or not a long list of articles should be "redlinked", and it's not like the entire database is searched to find articles that belong to the category; that information is pre-indexed by the database server, right?
I would feel uncomfortable proceeding with mass list-to-category conversion if the developers thought it would negatively impact site performance in a significant way.  Jamesday seems to think this problem will be fixed in 4-9 months.
The next most important problem is the ability to watch and examine the history of categories.  I think people should be able to veto conversion if they are watching a list or would like the history to remain viewable in a central location.  We should also file a feature request in bugzilla so that these issues will be fixed, and then later come back and convert the lists that people were watching into categories.  I don't find the "move it to a talk page and have a bot sync it" solution to be particularly satisfying.  The server load generated by such a bot would be quite high, especially given the performance hit of each category load.  The sheer number of lists that would have to be synchronized with categories would mean that updates would be significantly delayed, probably on the order of days.  I think it would be better just to leave the original lists in place for the time being.  If someone wants to run a bot to temporarily synchronize existing, redundant categories and lists, that might be a reasonable interim measure, but it might be easier just to wait for code improvements so we can do full conversion.
Ease of use is also an important issue - if it's hard to make useful contributions, fewer people will do it.  If you could edit the category membership by editing the category page (instead of 700 individual article pages), that would be a lot easier.  But personally, I wouldn't worry about getting this fixed before starting mass conversion.  Especially since redundant lists create opportunities for editors to waste time making redundant improvements.
 
There's also question about whether or not every single list that an article is on should have a link from the article itself.  Many people seem to find that doing that would unnecessarily clutter the navigation area at the bottom of the article, and that people should look at the "What links here" page to find all the lists that point to the article.  Personally, I find this unsatisfactory, because it's not obvious to the new user that this information is available through this link, and also because there are a lot of other inbound links that clutter up that page.  (Including many user and talk pages, and other articles which have common list or category memberships.)  On the other hand, it would be nice to segregate the "important" memberships from the "trivial" ones, for easier navigation.
One way to do this would be to mark categories as "incidental" or "vital", and have two navigational boxes, as appropriate.  Another would be to continue to use lists for "trivial" or "incidental" facts, and use categories only for "vital" memberships.  (Or "topical categories" vs. "lists of things with certain attributes", etc.) A bot could be used to make sure that each article has a "See also" link for each list it is a member of.  Alternatively, we could just convert lists of both kinds into categories, and just make a convention that the "important" ones should be first in the list.
All of the other negatives of categories can also be fixed with code improvements, including the ability to have manually sorted subsections. (Well, I don't actually care about broken mirror sites.)
-- Beland 03:21, 30 Mar 2005 (UTC)
thread_title: Weighing pros and cons

message: Honestly, I think the thing to do right now is to file a bunch of requests in bugzilla, and then put this project on hold for a few months until some critical fixes can be made.  (And perhaps lobby a little for developer attention, so the project can proceed as quickly as possible.)  It might be OK to deal with lists which are redundant with existing categories, to prevent wasted effort.  But we'll need to get an idea from the developers about how many conversions would be a reasonably safe amount, and then have some process for allowing people to object to conversion for change-watching reasons.  (Otherwise, I'm sure we wouldn't get community approval to run a bot for this purpose.) I'm not sure it's worth the effort of arguing about which lists are most important to convert, if we're going to be able to open the gate much more widely in a few months anyway.
In fact, I think the interim might be better spent converting "See also" links into categories.  (Which is actually sort of what was done in the "Gay icon" case.)  There's a lot more manual involvement here, and a lot of immediate benefits, including reduced redundancy and increased maintainability of cross-references, and considerably improved navigation.  The performance impact should also be somewhat less, both because things will proceed more slowly, and because the categories already exist.
The only navigational penalty for converting "see also" links into categories is the additional click and page load that's required.  But that's what navboxes are for.  Adding a navbox to a category automatically adds all the articles that contain that box to the category, so there's a neat solution which satisfies both the need for short click-paths and easy navigation.
(That makes me wonder whether category "peers" shouldn't automatically be displayed at the bottom of every article.  But that seems like a question for another page and another day.)
In any case, I've done some automated scans of previous database dumps to find articles in most need of "see also" link conversion.  There are some subpages of Wikipedia:Auto-categorization which present the results in a convenient report designed to put everything a human editor needs to the conversion for a given article in one place. -- Beland 03:21, 30 Mar 2005 (UTC)
thread_title: Next steps

message: I can implement any bot requests related to this project, if no one else is interested.  Only fairly trivial enhancements would be necessary for Pearle to be able to do this.  (I could also pass the source code on to someone else who would actually operate the bot.) -- Beland 03:21, 30 Mar 2005 (UTC)
thread_title: Bot

message: Lists have one vital feature that categories cannot yet have: they can have entries for articles that do not exist. In addition, lists can support semantics that categories cannot. I believe that the proposals here are over-hasty, and are getting ahead of what the category scheme can as yet support semantically. In any case, the lists should be kept as long as possible until there is general community consensus for how to proceed. -- The Anome 07:44, Mar 30, 2005 (UTC)
:Do you think my proposal addresses your concerns?  Thanks! --Yurik 08:02, 30 Mar 2005 (UTC)
replies: text: Anome, no one worth listening to thinks we should replace ALL lists with categories. That's why I moved this to "merge some redundant lists to categories". Philwelch 03:23, 31 Mar 2005 (UTC)

text: If a list is redundant with a category, the list should be made more useful, such as with annotation or more useful sorting/grouping. Stan 01:38, 2 Apr 2005 (UTC)
replies: text: What if someone changes the list or adds or removes an article from the category? We have subcategories for sorting and grouping, Stan. What we shouldn't have is two identical sets of redundant, 'non-identical' information. And that will inevitably happen whenever a list is redundant with a category. Philwelch 04:21, 2 Apr 2005 (UTC)
thread_title:  Please don't do this yet! 

message: Is'nt this problem a problem of semantic structuring? Lists and categories have something in common - they comprise of a list of items which are hierachically linked.
=== So what is the difference? ===
* Categories usually do not link items by an order other then alphabetical (it's possible though, but it is a pain) Lists primarily do that
* the formatting of lists gives mor freedom
* Categories live bottom up, lists top down (though categories can do something that lists can)
* Pointers versus Edges - Categories use edges, while lists only use pointers. The concept of edges implies automatic handling of "the second side" of the link. (a new category is created only with a new article -> the category page has to be maintained) - the other way around it is not possible (currently).
* orphaned links can be created by lists, but not by categories (currently)
* readers often want to know the semantics of a lemma and browse through equivalents or antonyms - tat's why navigation bars are so popular
=== What we need ===
What we actually want to achieve is a semantic structure, where we try to use non primitive constructions, without having the primitives integrated yet (Neither categories, lists or navifgation bars are). We need some basic primitives to link articles in a structured mannor. 
* External structuring information: (already done with categories) "element of" <-> "category"
* Properties: "name of property: value" (currently not well solved) - to make automatic structured list generation possible as well as defining proper prede- and successors
* list formatter: The lists associated with categories should be formatable by specifying a template
* navigator: a navigation bar, that displays some prede- and successors and the category, also with an adjustable template (format probably like the category entry + template link)
* semantic browser: a browser that can read the data and display it like the visible thesaurus or freemind would be really great
* structuring: It is already demonstrated in Wikipedia:Wikispecies, well we do not have to go that far but the goal would be something like OWL for semantics
--BoP 22:29, 30 August 2005 (UTC)
thread_title:  Semantic structure?! 

message: What we actually want to achieve is a semantic structure, where we try to use non primitive constructions, without having the primitives integrated yet (Neither categories, lists or navifgation bars are). We need some basic primitives to link articles in a structured mannor. 
* External structuring information: (already done with categories) "element of" <-> "category"
* Properties: "name of property: value" (currently not well solved) - to make automatic structured list generation possible as well as defining proper prede- and successors
* list formatter: The lists associated with categories should be formatable by specifying a template
* navigator: a navigation bar, that displays some prede- and successors and the category, also with an adjustable template (format probably like the category entry + template link)
* semantic browser: a browser that can read the data and display it like the visible thesaurus or freemind would be really great
* structuring: It is already demonstrated in Wikipedia:Wikispecies, well we do not have to go that far but the goal would be something like OWL for semantics
--BoP 22:29, 30 August 2005 (UTC)
thread_title:  What we need