Wikipedia talk:Database download/Archive 1

message: Hi, is there an update of wiki2static script which converts the sql-dump to a html file structure? This script has problems with conversion of the wiki table definitions to html. For this reason, articles containing tables aren't readable.
:Hi, I haven't worked on the script for a while, so I didn't put the new table syntax into wiki2static. I'll see if I can do it in the following days. Alfio 14:58, 16 Mar 2004 (UTC)
thread_title:  wiki-table to html-table problems in script wiki2static 

message: I have the Spanish version in a SQL database and the old_text entries are a bunch of symbolic nonsense.  For instance, the entry for Andorra is 
"UAN1E÷•¸ƒ×¨*g@€`›Š]Õ…;q[£ÄŽìÑ#³D\O;@Ê"þqÞÿÿ¾J#(}8*4¦RªîŒàÈž1á2BQiø¥Žp+IÍBun=ž²:T6R_ÂaPq:tN
]Dî2A5õJÆ	$òD#z\”à:7¢HbÌ€Ðß?=3´nìEWðŒù!a.ZGCŒåÑ
/U¸>œËE¢XðÔÃçâºÇÜÎw.µ“7õÕÕbçz³¹yâŒq»„GÒ€íÿ)þtÇd¢³ð0e›…;b<àÀ*ä³ü½ªýa½pæ†6ÏkÊÓ/ ¨)oQþBø"
I don't know if this is related, but when I loaded up the .sql file I got the following error:
"ERROR 1064 at line 269: You have an error in your SQL syntax..."
:It's compressed text, I would guess. See the section "Format Change" at http://download.wikimedia.org/ Mr. Jones 12:55, 9 Aug 2004 (UTC)
thread_title:  Compressed text? 

message: Hmmm. I'm having some trouble with this using python (specifically, ipython). 
import MySQLdb as m
conn=m.connect("localhost", "user", "password")
conn.select_db("wikipedia")
conn.query("SELECT * FROM old WHERE old_user=333")
r=conn.store_result()
c=[0]
while (c[len(c) - 1] != ()):
    c.append(r.fetch_row())
z=c[len(c) - 2][0][3]
import gzip
gzip.zlib.decompress(z)
---------------------------------------------------------------------------
error                                     Traceback (most recent call last)
/home/mrjones/<console>
error: Error -3 while decompressing data: incorrect header check
f=file("temp.gz", "w")
f.write(z)
f.close()
del f, z, c, r, conn, m
^D
Do you really want to exit ([y]/n)? y
host:~$gunzip temp.gz
gunzip: temp.gz: not in gzip format
host:~$
Mr. Jones 14:24, 16 Dec 2004 (UTC)
thread_title:  Problems with Python 

message: I tried importing the en.sql db into MySQL 4.1.2-alpha and got this error:
$ mysql wiki -u root -p < en.sql
Enter password: *******
ERROR 2013 at line 13147: Lost connection to MySQL server during query
Ran it again, got the same error after about 20 GB had been processed.
--
If I remember correctly, these errors are because the data that you are importing is sent over the connection in packets. Typically, in the server, you configure a limit to the size of these packets to avoid denial of service conditions. However, by doing that, you are also limiting the amound of data that can be put in one field, because each field has to be sent in one packet. It could be that one wikipage contains so much data that the server thinks you are trying to cause a denial of service, therefore it kicks you off.
The same thing happens when you are running mozilla's bugzilla; the packet size limits the size of attachments that users can insert into the database.
Tinus 22:19, 10 Aug 2004 (UTC)
thread_title:  Lost Connection Error using MySQL 

message: Is this information on Meta somewhere?  
It looks as though this page was once at m:Meta:Database download (back when the Meta: namespace there was called Wikipedia: also)... but it's gone now.  +sj+ 05:38, 16 Jul 2004 (UTC)
replies: text: I don't think it was on Meta. Links on Meta to Wikipedia:Database download will lead to this page via a redirect from Database download. Angela. 07:20, 16 Jul 2004 (UTC)
thread_title:  meta? 

message: for some odd reason the arrow on the link http://en.wikipedia.org/wiki/Wikipedia:Database_download in the page is not showing properly. i think it's something to do with the  and the wiki stylesheet, or maybe some extra bug in internet explorer... Vbs 09:03, 23 Jul 2004 (UTC)
thread_title:  Sample blocked crawler email 

message: How big is the uncompressed 20040811_old_table.sql.bz2 from en? Less than 30gb, I would guess, given that it was reported to be 18gb fairly recently, and starts out at about 9gb. What is the compression ratio? It seems that the suggested bzip2.exe for windows (used with XP pro) does not work (it's much slower than unxutils' one, which does the same thing, i.e. produces a file of over 30gb without stopping, it just does it faster).  I'll have to see if I can do it under Debian.   Mr. Jones 21:56, 13 Aug 2004 (UTC)
thread_title:  How big is the uncompressed wikipedia? 

message: Is there anywhere I can download the dump files that have been compressed with something other than bz2!? maybe gzip or zip?
: Why would you want to do that? (Please sign your posts with ~~~~) Mr. Jones 20:48, 16 Dec 2004 (UTC)
thread_title:  Database Dump Compression Format 

message: If someone gets to this, please include the different language wikinews databases in the DB dumps. -- Ilya 07:51, 23 Dec 2004 (UTC)
thread_title:  Wikinews database? 

message: Is there some reason that the weekly dumps have not been updated in three weeks? Michael L. Kaufman 16:06, Jan 26, 2005 (UTC)
:They are doing a database conversion I believe. It has been going on for quite some time. --Alterego 17:44, Jan 26, 2005 (UTC)
::There are new dumps up now from 2005-02-09, but unfortunately they seem to exclude "de" and "en" (and possibly others that I didn't notice). So if you're looking for those languages, it looks like you're still out of luck. -- All the best, Nickj (t) 01:04, 11 Feb 2005 (UTC)
replies: text: OK, I'm confused ... if you look here, there is a current en dump. It's not listed on the download page though. Does that mean that they should be listed and it's just a mistake that it's not, or does it mean that there's something wrong with that dump and that it has been excluded for a reason? The same thing applies to de. -- All the best, Nickj (t) 01:14, 11 Feb 2005 (UTC)
thread_title:  Dumps disabled? 

message: I'm trying to import the OLD data of the hebrew wikipedia (he) but it seems that because of the compression there are some special chars in the text (such as -  ' etc. ) that confuse the sql - which returen tons of errors. it seems to me that someone forgot formating the text (adding / before special chars and stuff) while exporting. (I've been importing the data few month ago on the same system with no problem, and the CUR data imports without any problem).
* does anyone else expirienced the same problem?
* can anyone check if it's not just my local problem?
* any idia about how to solve it?
thnx, Costello 19:44, 24 Feb 2005 (UTC)
thread_title:  Problems importing the compressed data 

message: Any chance of adding the imagelinks table to the dump script, too? It's a quite small table, and since links, categorylinks, brokenlinks etc. are getting dumped already, imagelinks is the only one still missing from a complete dump set.
Yes, I know that refreshlinks.php can be used to generate imagelinks - but it takes an *extemely* long time to run and also tends to bomb out on my installation eventually.
I've also tried to rework the obsolete rebuildlinks.php script for imagelinks only, but couldn't get it to work for images included by templates etc. and finally had to give up on this. I need imagelinks to decide which images to include in a notebook installation - more precisely, to drop (or not to mirror) orphaned images which are not referenced by a NS_MAIN or NS_TALK article (a considerable percentage BTW). --DerHund 14:14, 27 Feb 2005 (UTC)
replies: text: I've added this to the script, so it'll be in the next public dump. I'm expecting to start that within the next few hours. Any other table requests from anyone? Jamesday 13:21, 8 Mar 2005 (UTC)
replies: text: Dump started. We may have to abort it or part of it if it kills the site excessively - still working on making it less disruptive. It's sharing the image server which is experiencing very high disk load. Expect it to take 12 or more hours to run.  Jamesday 07:56, 9 Mar 2005 (UTC)
replies: text:  Suspended while processing de wikipedia because it was hurting the site too much. Will resume after peak time today. Jamesday 14:04, 9 Mar 2005 (UTC)
thread_title:  imagelinks table 

message: Production of new image tarball dumps has been temporarily suspended while we work on preventing the production of them from taking the whole site down. Will probably be back again within a few weeks. Jamesday 13:21, 8 Mar 2005 (UTC)
thread_title:  Image tarball dumps 

message: Whenever I try to import the sql dumps using phpmyadmin, it says No SQL Query
AxyJo 23:59, 17 Mar 2005 (UTC)
replies: text: I may be wrong, havong only used phpmyadmin a few times, but I don't believe you will be successful importing a large dump through it. --Alterego
How would I then import the dumps? 70.49.148.112 21:03, 19 Mar 2005 (UTC)
thread_title:  No SQL Query 

message: The concatenation of the dump files (english version of Wikipedia) has ended up with a file of around 32 gigs.
Apparently, the compression format has changed for bzip2 does not recognize the resulting file as a bz2 one but gunzip is able to uncompress the file (by naming the compressed file old_table.sql.gz).
Can anyone officially confirm the change in the compression format? 
Moreover, the uncompressed file has a size of only 34,201,462 KB which is not much bigger than the compressed file. Is that normal?
Nonetheless, the resulting sql files seems to be readable for it is possible to import the 'old table' from it. But I don't know whether the file is complete or not, and whether the old table that I got, will not miss any record.
Does anyone have similar problems?
--Kevouze 14:44, Apr 25, 2005 (UTC)
As of time of this post, the dump files now have the extension .sql.gz.  Does that mean that gzip should now be used to decompress them, and that the section on bzip2 is irrelevant? Tsointsoin 00:47, 30 July 2005 (UTC)
thread_title:  Compression Format Change and Size Issues 

message: The database dump hasn't been updated in a month now (it says it's done twice a week on the meta-wiki).
Image dumps have been broken for 2 weeks' time now. Image dumps use some strange compression that apparently can only be uncompressed using the right version of the right set of programs on the right platform (in other words, anything but standard platforms).
Is there any effort to standardize these practices, or is it always going to be "Whenever someone gets around to it or feels like doing it?" Is there anything Joe User can do to help out??
-- Q2 03:52, 16 Jun 2005 (UTC)
thread_title:  Database dumps old, image dumps gone 

message: I'm running stock Redhat 9 and can't seem to get the 16 gig image dump to download.  I've tried both wget and curl but they both appear to have a 2gig file limit.
Has anyone had any success getting these to download on linux?  If so what did you use?
Thanks.
TheLoneCoder 02:30, 1 August 2005 (UTC)
thread_title:  Downloading Large Files on Linux 

message: I've downloaded and ungzipped the latest available dump at the time of this posting (20050623_cur_table.sql). Upon importing it using "mysql -p -u root wikipedia < 20050623_cur_table.sql", I got an error and the import was not completed correctly:
ERROR 1062 (23000) at line 1488: Duplicate entry '8-VfD-Q+ç?' for key 2
According to "select count(*) from cur;", I only got 606,328 entries in the table.
I was able to work around the problem by editing the text file 20050623_cur_table.sql by hand, and removing the "UNIQUE" in front of the key 'name_title_dup_prevention' (for info HexToolbox is the only text editor I found capable of editing such a large file).  The import then gave me 1,811,554 entries.
Am I the only one getting this error?  Is there any better solution than this workaround?
I am on Windows XP using MySQL 14.9 Distrib 5.0.3-beta
Tsointsoin 17:09, 2 August 2005 (UTC)
thread_title:  Error "Duplicate entry '8-VfD-Q+ç?' for key 2" on import 

message: There was a new dump of the en.wikipedia cur table this weekend, and I'm itching to get my hands on it. Unfortunately for me and my software, the dump is in the new XML format rather than an SQL query. Is there a tool, perhaps, for importing the XML dump into mysql? — brighterorange (talk) 01:50, 7 September 2005 (UTC)
thread_title:  XML import 

message: Moved from Wikipedia:Village pump on Thursday, July 10th, 02003.
I wonder if someone thought about making dict files of the Wikipedia. It would be cool to have the Wikipedia wherever I am, independent of an internet connection. (Okay, I still need my laptop for this...) dict seems a good way to achieve this. I'm willing to spend some time hacking a Python script that can create the dict files from the SQL stuff. But I'd like to know if other people are interested in this as well, or maybe there's someone who already did this job... :) --Guaka 22:38 5 Jul 2003 (UTC)
: Doesn't Tombraider achieve this? CGS 22:40 5 Jul 2003 (UTC).
Dear Wikipedians!
I enjoy very much my tomeraider wikipedia edition from december 2003. And I dream of downloading a current version. At that time it was 180 mb with 180000 articles. Now there are 360000. Please !!!! Thousands of PDA friens will be grateful to you !
The german wikipedia for tomeraider is available for download from 1 of september 2004 with 217 mb and 180000 articles. Vlad 
:: You mean Tomeraider? No... First of all, tomeraider is shareware. And AFAICS it is totally not meant to convert the wikipedia into the dict format. Guaka 02:37 6 Jul 2003 (UTC)
::: Ha ha :) I know it's not meant to convert files to dict format, but it does what you want - view files on the go without a net connection. CGS 20:28 6 Jul 2003 (UTC)
:::: Another thing is... Tomeraider is non-free software. This is already enough reason not to use it. But even if I wanted to, I couldn't because I run GNU/Linux. Guaka 16:06 7 Jul 2003 (UTC)
::::: If it's the right tool for the job, swallow your pride and run it through Wine. CGS 22:15 7 Jul 2003 (UTC).
::::I guess you paid for the PDA hardware. So why is $20 for good software a no-go? I chose TomeRaider because it was the best option at the time (and it may still be, not sure). Some people write software for a living, and if they are good at it, I hope they continue doing so. Just because they earn some money they are not neccesarily a second Bill. Not that I wouldn't prefer GNU software which is equally cross plaform, fast and economical with PDA space, it just doesn't seem a matter of higher principle to me. Erik Zachte 22:54, 18 Mar 2004 (UTC)
replies: text: I fully agree with that Nelson. We just need to have a name now, so that we have a page for it. Or maybe this project would better fit on the Meta Wikipedia? Guaka 00:10 10 Jul 2003 (UTC)
Try the following Perl script for generating the Dict database.  Change the DBI->connect to have the correct values for username and password instead of dbuser and dbpass.
 #!/usr/bin/perl -w
 
 use strict;
 use DBI();
 
 sub article2dict {
   my ($title, $text) = @_;
 
   $title =~ s/_/ /g;
   $text =~ s/\r//g;
   $text =~ s/^/  /mg;
 
   print "$title\n";
   print $text;
   print "\n\n";
 }
 
 # Connect to the database.
 my $dbh = DBI->connect("DBI:mysql:database=wikipedia;host=localhost",
 		       "dbuser", "dbpass",
 		       {'RaiseError' => 1});
 
 # Now retrieve data from the table.
 my $sth = $dbh->prepare("SELECT cur_title, cur_text FROM cur " .
 			"WHERE cur_namespace = 0 ORDER BY cur_title");
 $sth->execute();
 while (my $ref = $sth->fetchrow_hashref()) {
   article2dict($ref->{'cur_title'}, $ref->{'cur_text'});
 }
 $sth->finish();
 
 # Disconnect from the database.
 $dbh->disconnect();
===wik2dict.py===
I finally wrote something: wik2dict.py. It tries to create reasonably layouted dict articles. It can also automatically fetch the database dumps. There are some requirements though. And currently it is only version 0.2. So beware.
I would appreciate it if someone (possibly someone at Wikimedia?) could run the script regularly and put the dict files available for everyone to download. Too bad they can't be included in Debian though ("GFDL is non-free").  However, the script itself could probably be included in contrib :) G-u-a-k-@ 17:50, 27 Jul 2004 (UTC)
thread_title:  Wikipedia in DICT format 

message: I finally wrote something: wik2dict.py. It tries to create reasonably layouted dict articles. It can also automatically fetch the database dumps. There are some requirements though. And currently it is only version 0.2. So beware.
I would appreciate it if someone (possibly someone at Wikimedia?) could run the script regularly and put the dict files available for everyone to download. Too bad they can't be included in Debian though ("GFDL is non-free").  However, the script itself could probably be included in contrib :) G-u-a-k-@ 17:50, 27 Jul 2004 (UTC)
thread_title: wik2dict.py

message: For those who would like to keep their local copy in sync, I would suggest to set up a CVSup server ( http://www.cvsup.org/ ). The snapshot can be made a number of time per day (say 4 times a day) and people can very efficiently synchronize
to the latest version of the wikipedia. This is a lot faster and saves a lot of bandwidth compared to downloading a complete tarball every time. The server does not even have to run on the wikipedia server itself, but it seems the logical choice.
CVSup is very efficient. If the wikipedia dumps can be tagged with a version number similar to RCS, the synching will probably be blazingly fast. -- Tim Hemel
:I don't think that would work very well with the dumps. The bzip2-compressed versions are not going to be cleanly diffable, and if I leave them as text (~380 megabytes for English current revisions, a few gigabytes for old revisions; I'm reluctant to have them sitting around uncompressed), they're still not going to work that well in CVS if I understand its storage system correctly. Each line of the dump is an SQL INSERT statement for about 500 pages, and the slightest change to any of them (including cache invalidation timestamps) would cause the whole line to be sucked out and replaced. --Brion 18:33 25 May 2003 (UTC)
::I'm not sure how applicable cvsup would be, but I think rsync is worth considering. Rsync doesn't use diff for computing deltas, so I don't think the "long lines with small changes" problem applies. As for rsyncing compressed files, there are techniques to do that, such as http://svana.org/kleptog/rgzip.html or http://lists.samba.org/archive/rsync/2002-October/004035.html (merely two results from a brief Google search, I'm sure more research on the topic would be fruitful). If people feel this would be worth pursuing, let me know. Neilc 11:30, 7 Aug 2004 (UTC)
replies: text: gzip does have an --rsyncable option (see http://rsync.samba.org/ftp/unpacked/rsync/patches/gzip-rsyncable.diff and gzip --help). bzip2 doesn't seem to. See http://www.debianplanet.org/node.php?id=524 for a discussion of pros and cons (for w:Debian). I don't know how relevant the server load problem is for d.wp . 
See also http://lists.debian.org/debian-devel/2001/10/msg02187.html 
About efficiency in part (reportedly):  http://rsync.samba.org/tech_report/
More later.
Mr. Jones 05:07, 14 Aug 2004 (UTC)
thread_title:  CVSup/Rsync, and why it would be useful 

message: I am conducting research which utilizes the content of wikipedia. I can access the content of my wiki database dump via a web browser and apache, but this does not suit the nature of my work. I would like to just access the plain (or marked up) text of pages through sql queries. How can I do this? I would imagine this requires 2 things, mapping of a page title to an id, and selecting the textual content associated with that ID. can any one please advise?  —Preceding unsigned comment added by 128.238.35.108 (talk) 02:05, 7 December 2007 (UTC)
thread_title:  Direct access to textual content of Wiki Pages via mySQL? 

message: Are the English Wikipedia images dump actually up to 75.5G from 15G in June?  Could this huge tar file be broken into 10G batches or something?  75G is quite a download. 
:Rsync is the best solution for downloading "upload.tar" file with images (which is 75 GB in size). It allows resuming download, automatic error checking and fixing (separately in any small part of file). And if you already have old version of "upload.tar" - it will download only the difference between them. But this "updating" feature of Rsync is useful only for TAR archives (because they only collect and don't compress data), and useless for files compressed with gzip/bzip2/7-zip, because any little change in data will cause great changes in almost whole file.
:For more info look here: http://en.wikipedia.org/wiki/Wikipedia:Database_download#Rsync
:--Alexey Petrov 03:26, 9 April 2006 (UTC)
replies: text: I thought that the images served by Wikipedia are already compressed, so their data is near random and thus gzip/bzip/7-zip/etc are useless on them?  —Preceding unsigned comment added by 67.53.37.218 (talk) 03:21, 11 May 2008 (UTC)
thread_title:  Unwieldy Images 

message: It seems to me that a lot of people download the database for the "Help:" Namespace, it would seem logical to me to provide a seperate download just for this. This could be done via the perl script that s provided in this very article. It would; 1, save bandwidth, 2, save time, 3, make life easier for mediawiki users :), 4 not be very hard. 
Otherwise, would anyone be able to point me in the right direction of someones own download of it?
:I would like to second this comment.  The idea of creating such a great tool and then forcing new administrators to have to download the entire database just to get the "Help:" pages is crazy Armistej
And one more "gotcha".  You need the "Template:" namespace too.  Otherwise a lot of your "Help:" pages are missing bits and pieces from them.  Unfortunately, the "Template:" namespace contains templates from all parts of the MediaWiki, not just templates relating to "Help:".  If I'm running an all-English MediaWiki on my Intranet, I couldn't give $0.02 about the Russian, Chinese, or other non-English templates which will never be referenced.  We need some sort of script to find the "dangling" templates and zap them once and for all.
:Exactly! I started a Help Desk query about this on 07/08/06, not having seen the above yet, nor having known to ask about the template. I'll repeat here that my motivation was that I do not currently have internet at home, so learning about wiki takes away from my short online opportunities. I bet others are in this boat.
::I am appalled that there still appears to be no simple and straightforward way to download the "Help:" namespace and/or the "Template:" namespace. Either that, or it's too difficult to find. Come on, the comment just above is over a year old! Anyone have a simple point-and-click solution yet? --Wikitonic 18:40, 24 September 2007 (UTC)
thread_title:  Help: Namespace download 

message: I think that it'll be precious when you'll do more frequent dumps of cur table.  Full dumps takes a lot of time and not everyone need them. As I see it was started yesterday and not finished yet. What do you think about making small dumps once a week and full once a month? 
Margospl 19:45, 10 Mar 2005 (UTC)
thread_title:  More frequent ''cur'' table dumps? 

message: Is there any way to save all my user contributions? That is, when I click on "my contributions" from my user page, I want to have all the links to "diff" saved. I wonder if there is a faster way to do this than click "save as" on each link. Q0 03:52, 7 May 2005 (UTC)
replies: text: Look into a recursive wget with a recursion level of one. --maru   (talk)  contribs 03:44, 19 May 2006 (UTC)
thread_title:  Saving user contributions 

message: * Thank you Beland for updating the page, and for making it clear that XML dumps are the way of the future.
* Is there a pre-existing way that anyone knows of to load the XML file into MySQL without having to deal with MediaWiki? (What I and presumably most people want is to get the data into a database with minimum pain and as quickly as possible.)
* Shouldn't this generate no errors?
 xmllint 20050909_pages_current.xml
Currently for me it generates errors like this:
20050909_pages_current.xml:2771209: error: xmlParseCharRef: invalid xmlChar value 55296
[[got:���������&
                                                                              ^
20050909_pages_current.xml:2771209: error: xmlParseCharRef: invalid xmlChar value 57158
����������
                                                                              ^
-- All the best, Nickj (t) 08:36, 14 September 2005 (UTC)
thread_title:  New XML database dump format 

message: I was trying to import the id languange wikipedia but the importDump.php stop at article 467 (from total of around 12000+). Anybody can help me with this problem ? Borgx(talk) 07:53, 16 September 2005 (UTC)
thread_title:  Problem with importDump.php 

message: Is there any way for importing the xml dump files successfully ? importDump.php stop uncompleted with no error. sql dump files is too old :( ,xml2sql-java from Filzstift only importing the "cur" table (I need all tables for statistical needs. Thanks Wanchun, but I still need to see the tables). Borgx(talk) 00:44, 21 September 2005 (UTC)
thread_title:  How to import the files? 

message: It might be worth commenting that importing the data from the new SQL dumps will be quicker if  the keys and indicies are disabled when importing the SQL. --Salix alba (talk) 23:22, 30 January 2006 (UTC)
replies: text: Check out meta:Talk:Data_dumps#HOWTO_quickly_import_pagelinks.sql.--Bkkbrad 15:51, 22 February 2007 (UTC)
thread_title:  Disable keys 

message: Maybe it says somewhere, but I just don't see it.Gil the Grinch 16:09, 21 February 2006 (UTC)
thread_title:  How do you retrieve previously deleted files? 

message: I have downloaded and decompressed a dump into an XML file. Now what? I made the mistake of trying to open that and it froze up my computer it was so huge! What program do I open it with in order to view the Wikipedia content? What next?! → J@red  01:08, 31 May 2006 (UTC)
replies: text: Yes I've had the same problem, never managed to successfully get a dump imported into SQL on my machine. It all depends on what you want to do, Wikipedia:Computer help desk/ParseMediaWikiDump describes a perl library which can do some operations. I've avoided SQL entirely, instead I've created my own set of perl scripts which extracted linking information and writes it out to text files, I can then use the standard unix grep/sed/sort/uniq etc tools to gather cetain statistics, for example see User:Pfafrich/Blahtex en.wikipedia fixup which ilustrates the use of grep to pull out all the mathematical equations. --Salix alba (talk) 06:50, 31 May 2006 (UTC)
replies: text: Well I understand that there are several ways to parse it, but is there any feasible way of easily opening up information from a dump in a program like IE or FireFox for viewing, just like I'm viewing this page now? → J@red  19:28, 1 June 2006 (UTC)
thread_title:  Opening the content of a dump 

message: I travel a lot and some times like to look up stuff on my laptop, I see that I can do it but like a lot of people I am clueless about computer was hoping that someone would make a section with  easy to follow steps to make this work figure that I’m not the only one who would like this .
I see the size of wiki with picture 90G wow little to big hehe but how hard would it be to just get picture for say featured article and the rest text only Britannica and Encarta are both 3-4 gigs. if it is text only would be fine.Ansolin 05:59, 14 June 2006 (UTC)
replies: text: That's basically what I want, too. → J@red  19:39, 15 June 2006 (UTC)

text: If you're running GNU/Linux you can give wik2dict a try. DICT files are compressed hypertext. wik2dict needs an upgrade though. I will work on that very soon. I have also written MaemoDict, to have Wikipedia on my Nokia 770, though I don't have any space on there to put a big Wikipedia on it (so one of the things I will add to wik2dict is to just convert some categories, instead of the whole thing).There might also be some software to support dict stuff in Windows. Guaka 20:28, 15 June 2006 (UTC)

text: See, the problem I have is that I've downloaded the wiki from the downloads page and I've decompressed it, but it's just a big gargantuan xml file that you can't read. How would I now, offline, open/dump that information in a wiki of my own? Do I need mediawiki? This is all so confusing and it seems that nowhere is a proper explanation! → J@red  12:36, 16 June 2006 (UTC)
thread_title:  Simple steps for wikipedia while you travel 

message: I've imported the text dump file into my local Wiki using importDump.php. And I've downloaded the huge tar image dump,and put the extracted files (directory intact) under the upload path specified in the LocalSetting.php. But my Wiki installation doesn't seem to recognize them. What do I need to do? I think it has something to do with RebuildLinks or RebuildImages or something else. But I want to know the specific procedures. Thanks! ----Jean Xu 12:26, 2 August 2006 (UTC)
thread_title:  How to import image dumps into a local Wiki installation? 

message: People seem to be having lots of issues with trying to get this working correctly without any detailed instructions, myself included.  As an alternative, there is WikiFilter which is a far simpler tool to setup and get working.
I am not affiliated with WikiFilter, I have just used the app and found it quite successful.
WizardFusion 08:59, 20 September 2006 (UTC)
thread_title:  Using XML dumps directly 

message: Is it possible for someone to create an official torrent file from the wikipedia .bz2 dumps? this would encourage leeching to some degree but it would make it easier for people to download the large files for offline use (i would like wikipedia as maths/science reference for offline use when travelling).
129.78.64.106 04:54, 22 August 2006 (UTC)
thread_title:  Possibility of using Bittorrent to free up bandwidth 

message: *Not everybody who wants to have access to Wikipedia offline is a computer systems expert or a computer programmer.
*What is with this bzip crap, why not use winzip.  I've spent so long trying to figure out this bzip thing, that I could have downloaded 10 winzip files already
*Even if i get this to work, can I get a html end product that I browse offline as if I was online?
— 195.46.228.21 12 September 2006.
Agree, wikipedia a very usefull site has failed the user friendly concept of offline browsing.
# Granted.
# The best reason not to use WinZip is probably that WinZip is not free.  On the other hand, bzip2 is.  It is also a better compressor than WinZip 8, although WinZip 10 does compress more.   And because the source code of bzip2 is freely available, if someone has a computer that cannot decompress bzip2 files, it is much easier to rectify this situation than in the proprietary case.
# It sounds like you want a Static HTML tree dump for mirroring.
# Remember: just the articles are about 1.5GB compressed (so maybe around 7-8GB uncompressed?).  Current versions of articles, talk pages, user pages, and such are about 2.5GB compressed (so I'd estimate about 13GB uncompressed); Images (e.g. pictures) will run you another 75.5GB.  If you also want the old revisions (and who wouldn't?) that's about 45.9GB (I guess around 238GB uncompressed).
— The Storm Surfer 04:08, 7 October 2006 (UTC)
replies: text: Try 7-Zip for a compression utility that supports many formats. Superm401 - Talk 09:00, 3 February 2007 (UTC)
thread_title:  Why is this all so complicated? 

message: I have written simple a page for getting the basics working on Ubuntu Linux (server edition). This worked for me, but there are issues with special pages and templates not showing. If anyone can help with this it would be great. It's located at http://meta.wikimedia.org/wiki/Help:Running_MediaWiki_on_Ubuntu.  WizardFusion 23:31, 1 October 2006 (UTC)
thread_title:  Possible Solution 

message: Hello, why are the image-dumps from last year? Would it be possible to get the small thumbnails as an extra file?
It should be possible to seperate the fair-use images from the dump. :de:Benutzer:Kolossos09:14, 12 October 2006 (UTC)
thread_title:  old images 

message: When is Wikipedia going to be dumped again? Salad Days 06:35, 30 December 2006 (UTC)
replies: text: It seems to be about monthly. Usally on the lower side Reedy Boy 13:12, 30 December 2006 (UTC)
replies: text: Where are the new dumps? The new place for dumps is also orphaned for more then a week. We at Wikipedia-World could need it really. Thanks.  Kolossos 17:54, 23 January 2007 (UTC)
replies: text:  The latest dump is also over a day beyond it's Estimated finish time.  Does this mean it failed?  Is there anyone who can comment on the actual status? —The preceding unsigned comment was added by 216.84.45.194 (talk) 21:50, 1 February 2007 (UTC)
replies: text: Looks like it did fail but a new one was started since Flamesplash 18:15, 10 February 2007 (UTC)
thread_title:  Update 

message: Rsync section seems to say that rsync will work at least as well with bzip2 as with gzip --rsyncable.  Not sure this is true.  bzip2 does compress using blocks, but (1) the bzip2 manpage says blocks are at least 100KB long, and (2) the blocks are on even 100kb boundaries, so any changes in the length of one article will affect the rest of the archive.  gzip --rsyncable seems to ensure that less than 100k of compressed output after a change is affected (I'm basing that the RSYNC_WIN constant of 4096 in the gzip patch; I figure that's just the order of magnitude).  More importantly, gzip uses a rolling checksum to decide where block boundaries go, in such a way that inserting or deleting bytes doesn't affect the location of all future block boundaries.  It's clever, and kind of out of my league to explain it clearly.
If CPU time on the wikimedia servers isn't an issue, my guess is that rsync -z on an uncompressed file is the lowest-bandwidth way to do incremental transfers, because changes are more localized than under other approaches and the network stream is still gzip-compressed (possibly at a lower compression level than gzip uses on files).  Would require more disk space, too.
Of course, really it comes down to 1) what differences between these approaches actually turn out to be in testing, and 2) whether there's demand for less bandwidth-heavy updates given the costs (or other things like how busy the wiki sysadmins are).  I'm not up to date on what discussions have already happened. —The preceding unsigned comment was added by 67.180.140.96 (talk) 00:32, 14 January 2007 (UTC)
thread_title:  Info on gzip --rsyncable vs. bzip 

message: For whatever it's worth, diffing XML dumps using rsync --only-write-batch -B300 and compressing the result with bzip2 seems to produce a file that's substantially smaller than the monthly dump.  (This is based on testing with ruwiki-20061108-pages-articles.xml and ruwiki-20061207-pages-articles.xml: rsync-batch.bz2 was 16.6 MB while the 20061207 dump was 79.9 MB bzipped.)  Producing dumps with only new and changed articles (and tools to process them) might also be useful.  Again, wikifolks may not have the time or the need for either.  Bigger gains may come from using smaller/larger rsync block sizes. 67.180.140.96 03:35, 14 January 2007 (UTC)
thread_title:  Followup on rsync/incremental transfer 

message: The 1-24-2007 dump looks to be stalled/broken.  It's several days past it's ETA, and in the past it has only taken a day or two.  Can someone comment on the actual status or if I should be notifying someone else through a different mechanism so that this can be restarted, or the next dump began. Flamesplash 17:02, 5 February 2007 (UTC)
thread_title:  1-24-2007 dump frozen 

message: The image dumps are over a year old now (last modified 2005-Nov-27), is there any plan to update them? Bryan Derksen 10:23, 15 February 2007 (UTC)
replies: text: What are you talking about...? Look Here 2007-02-07... Reedy Boy 10:57, 15 February 2007 (UTC)
replies: text: There are only database dumps for the various SQL tables at that URL. Image file dumps are located at http://download.wikimedia.org/images/wikipedia/en/ and were last updated in November 2005, as Bryan wrote. I'd also be interested in a newer version.--134.130.4.46 23:54, 22 February 2007 (UTC)
replies: text: The image metadata database does get dumped so if all else failed I suppose one could rig up a script to download the matching images directly from en.wikipedia.org. I imagine that would take longer and put more of a load on the servers than an image dump would, but if that's the only place the images are available then that's all that I can think of doing. Bryan Derksen 01:26, 1 March 2007 (UTC)
thread_title:  Image dumps getting old 

message: Not sure where to put this, kept looking for a mail add to some tech staff. Anyway, according to the logfile on http://download.wikipedia.org, the December dump is in progress, and the link points to the November one. However the url http://static.wikipedia.org/downloads/December_2006 works perfectly, so I guess all that needs doing is updating the link? (http://houshuang.org/blog - pre-alpha tool to view html-dumps without unzipping) Houshuang 06:13, 21 February 2007 (UTC)
thread_title:  Link to HTML dumps not updated 

message: Wikisign seems to be down. Is there an alternative? MahangaTalk 02:53, 12 April 2007 (UTC)
thread_title:  Wikisign.org alternative 

message: The page says, that beginning with V1.5 there are routines to dump a wiki to html. How does this work? How can I use this on my own mediawiki? --Sebastian Dietrich 10:32, 19 May 2007 (UTC)
thread_title:  How do HTML dumps work?