Wikipedia Diskussion:WikiProjekt Georeferenzierung/Hauptseite/Wikipedia-World/news

message: The CSV file linked from this template is, as of 2006-12-29, encoded in a curious way. First, its original content was encoded in UTF-8: secondly, it was then decoded to Unicode as if the resulting byte-stream was encoded in Windows-1252: finally, the resulting Unicode was re-encoded as UTF-8. Finally the file was compressed in .zip format. 
After unzipping, applying the following UNIX command-line filter program in Python will remove these layers of encoding, and output a valid UTF-8 encoding of the original Unicode content of the CSV file. I hope this is useful.
import sys, string
def explodify(ch):
    if ord(ch) in [0x81, 0x8d, 0x8f, 0x90, 0x9d]:
       return chr(ord(ch))
    return chr(ord(ch.encode("windows-1252")))
# Stuff has been multiply encoded: remove these layers
while 1:
    text = sys.stdin.readline()
    if not text:
      break
    # Decode as UTF-8
    text = text.decode("utf-8")
    # Rip off a layer of Windows-1252 -> Unicode
    text = string.join([explodify(ch) for ch in text], "")
    # What is left is a valid UTF-8 string: which is what we want
    sys.stdout.write(text)
Note: even after running this decoder, a considerable number of HTML entity escapes, both numeric and otherwise, and %XX urlencode()-style hex escapes remain in the source data fields. However, these are easily dealt with without any detective work on the encoding of the file, and can be dealt with after CSV decoding of the valid UTF-8 data produced by this filter.
-- The Anome 19:16, 29. Dez. 2006 (CET)

message: CREATE TABLE IF NOT EXISTS `pub_C_geo_id` (
  `ID` int(9) NOT NULL AUTO_INCREMENT,
  `lang` varchar(10) COLLATE utf8_bin NOT NULL,
  `Titel` varchar(180) COLLATE utf8_bin NOT NULL,
  `Titel_en` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_de` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_es` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_fr` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_it` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_ja` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_nl` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_pl` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_pt` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_ru` varchar(200) COLLATE utf8_bin DEFAULT NULL,
  `Titel_sv` varchar(90) COLLATE utf8_bin DEFAULT NULL,
  `Titel_fi` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_no` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_eo` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_sk` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_da` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_cs` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_tr` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_zh` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_ca` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `Titel_is` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  `lat` double NOT NULL DEFAULT '0',
  `lon` double NOT NULL DEFAULT '0',
  `type` varchar(10) COLLATE utf8_bin NOT NULL,
  `pop` double NOT NULL DEFAULT '0',
  `Height` double NOT NULL DEFAULT '0',
  `Country` varchar(10) COLLATE utf8_bin NOT NULL,
  `Subregion` varchar(10) COLLATE utf8_bin NOT NULL,
  `Scale` varchar(10) COLLATE utf8_bin NOT NULL,
  `dim` varchar(10) COLLATE utf8_bin NOT NULL,
  `psize` double NOT NULL,
  `style` varchar(12) COLLATE utf8_bin NOT NULL,
  `t` varchar(10) COLLATE utf8_bin NOT NULL,
  `image` varchar(160) COLLATE utf8_bin NOT NULL,
  `imagejpg` varchar(160) COLLATE utf8_bin NOT NULL,
  `name` varchar(160) COLLATE utf8_bin DEFAULT NULL,
  PRIMARY KEY (`ID`),
  UNIQUE KEY `Titel_en` (`Titel_en`),
  UNIQUE KEY `Titel_es` (`Titel_es`),
  UNIQUE KEY `Titel_fr` (`Titel_fr`),
  UNIQUE KEY `Titel_it` (`Titel_it`),
  UNIQUE KEY `Titel_ja` (`Titel_ja`),
  UNIQUE KEY `Titel_nl` (`Titel_nl`),
  UNIQUE KEY `Titel_pl` (`Titel_pl`),
  UNIQUE KEY `Titel_pt` (`Titel_pt`),
  UNIQUE KEY `Titel_ru` (`Titel_ru`),
  UNIQUE KEY `Titel_sv` (`Titel_sv`),
  UNIQUE KEY `Titel_fi` (`Titel_fi`),
  UNIQUE KEY `Titel_no` (`Titel_no`),
  UNIQUE KEY `Titel_eo` (`Titel_eo`),
  UNIQUE KEY `Titel_sk` (`Titel_sk`),
  UNIQUE KEY `Titel_da` (`Titel_da`),
  UNIQUE KEY `Titel_cs` (`Titel_cs`),
  UNIQUE KEY `Titel_de` (`Titel_de`,`name`),
  UNIQUE KEY `Titel_tr` (`Titel_tr`),
  UNIQUE KEY `Titel_zh` (`Titel_zh`),
  UNIQUE KEY `Titel_is` (`Titel_is`),
  UNIQUE KEY `Titel_ca` (`Titel_ca`),
  KEY `Pop` (`pop`),
  KEY `type` (`type`),
  KEY `Character` (`psize`),
  KEY `lang` (`lang`),
  KEY `lon` (`lon`),
  KEY `lat` (`lat`),
  KEY `image` (`image`),
  KEY `Country` (`Country`),
  KEY `Subregion` (`Subregion`),
  KEY `Titel` (`Titel`),
  KEY `imagejpg` (`imagejpg`),
  KEY `name` (`name`),
  KEY `dim` (`dim`)
 ) ENGINE=InnoDB  DEFAULT CHARSET=utf8 COLLATE=utf8_bin  ; 
--Kolossos 19:09, 10. Feb. 2010 (CET)
thread_title:  Database- Structure