Ispygames

== The News ==
IGN hit with layoffs, 1UP, UGO and GameSpy shutting down
1UP, UGO and GameSpy to be shut down
Goodbye, And Thank You From The GameSpy Team
== The Problems ==
* Once you start digging around these sites you find it to be a mess of inconsistent url schemes and content everywhere. 
* Some files are being hosted on MediaFire.
* Based on tests the larger and older a site is the more that is missed by a wget crawl due to the url scheme. 
== What we know ==
* We already have a list of almost all the domains involved
* A clean list with dups and bad domains is already being process and will be posted here when complete.
* Most of the sites are not that big, but a few are huge.
== The plan ==
* Save the sites and related content
* Backup the Twitter feeds for any associated accounts. All my tweets just takes a username and returns the max tweets possible.
== wget test command ==
This if for the gamespy sites.
USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://planetdoom.gamespy.com"
WARC_NAME="warc_name"
wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST" \
--span-hosts --domains=$SAVE_HOST,pcmedia.gamespy.com,pnmedia.gamespy.com,pspmedia.gamespy.com,oystatic.ignimgs.com
Try this for the ign, ugo sites.
USER_AGENT="Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)"
SAVE_HOST="http://ve3d.ign.com"
WARC_NAME="warc_name"
wget -e robots=off --mirror --page-requisites \ 
--waitretry 5 --timeout 60 --tries 5 --wait 2 \
--warc-header "operator: Archive Team" --warc-cdx --warc-file="$WARC_NAME" \
-U "$USER_AGENT" "$SAVE_HOST"
== IGN domains ==
=== In progress ===
* http://aavault.ign.com - grabbed, checking for completeness
* http://autoassaultvault.ign.com/ - grabbed, checking for completeness
* http://beacon.snowball.com - already closed
* http://bombergirl.ign.com - closed contest page in flash
* http://o.cubemedia.ign.com - dead end
* http://actionunleashed.ign.com - grabbed, checking for completeness
* http://dndvault.ign.com - grabbed, checking for completeness
* http://vault.ign.com - grabbing
* http://ac2vault.ign.com - Smiley - done
* http://acvault.ign.com - Smiley - done
* http://aion.ign.com - Smiley Grabbing
* http://wowvault.ign.com - Smiley grabbing
* http://atvault.ign.com - Smiley grabbing
* http://alli.ign.com - Smiley done
* http://antis.ign.com - Smiley grabbing
* http://doa.ign.com - Smiley grabbing
* http://aovault.ign.com - siliconvalleypark - grabbed
* http://aevans.dev.m.au.ign.com - dead
* http://aevans.dev.m.ca.ign.com - dead
* http://aevans.dev.m.ie.ign.com - dead
* http://aevans.dev.m.ign.com - dead
* http://aevans.dev.m.uk.ign.com - dead
* http://cdyi.dev.m.ign.com - dead
* http://ve3d.ign.com - grabbed, checking for completeness
* http://flashpoint.ign.com - siliconvalleypark - grabbed
* http://beaterator.ign.com - siliconvalleypark - grabbed
* http://aivlev.dev.m.au.ign.com - password protected dev site
* http://aivlev.dev.m.ie.ign.com - password protected dev site
* http://aivlev.dev.m.ign.com - password protected dev site
* http://aivlev.dev.m.uk.ign.com - password protected dev site
* http://apassey.dev.m.ign.com - password protected dev site
* http://bestof.ign.com - Tracer grabbing
* http://www.blockbuster.ign.com - already dead
* http://kaneandlynch.ign.com/ - flash based
* http://wishvault.ign.com - siliconvalleypark grabbing, Smiley done
* http://witchervault.ign.com - siliconvalleypark, Smiley - grabbed, done
* http://www.supersmashbros.ign.com - Smiley grabbing
* http://www.championshipgamingseries.com - Smiley grabbing
* http://code.ign.com - Smiley grabbing
* http://mevault.ign.com - siliconvalleypark grabbing
* http://beacon.ign.com - Smiley, done
* http://blockbuster.ign.com - 404 not found
* http://broadband.ign.com - Smiley
* http://browserthemes.ign.com - Smiley
* http://ces2009.ign.com - Smiley, all broken links
* http://championshipgamingseries.ign.com - Smiley
* http://championsonline.ign.com - Smiley
* http://cohvault.ign.com - Smiley
* http://comiccon.ign.com - dead
* http://corp.ign.com - Smiley
* http://covvault.ign.com - Smiley, redirects to cohvault
* http://adtools.ign.com - blank
* http://crossassault.ign.com - Smiley, done
* http://design.ign.com - Smiley, done
* http://dragonica.ign.com - Smiley, done
* http://eq2vault.ign.com - Smiley, done with some timeouts
* http://eqvault.ign.com - Smiley, finished with some timeouts
* http://esports.ign.com - Smiley, done
* http://evevault.ign.com - Smiley
* http://feeds.ign.com - Smiley, done
* http://ffvault.ign.com - Smiley, finished with some timeouts
* http://findit.ign.com - Smiley, done
* http://gamechanger.ign.com - Smiley, done
* http://gamesites.ign.com - Smiley, done
* http://gamestore.ign.com - Smiley, done
* http://gamingworldrecord.ign.com - Smiley, done
* http://gbartone.dev.m.au.ign.com - not found
* http://gbartone.dev.m.ca.ign.com - not found
* http://gbartone.dev.m.ie.ign.com - not found
* http://gbartone.dev.m.ign.com - not found
* http://gbartone.dev.m.uk.ign.com - not found
* http://grandtheftautohood.ign.com -> grandtheftauto.ign.com
* http://grandtheftauto.ign.com - Smiley, done
* http://gtahood.ign.com -> grandtheftauto.ign.com
* http://gta.ign.com - Smiley, done
* http://www.blockbuster.ign.com - already dead
* http://guitarhero3.ign.com - Smiley, done
* http://gwvault.ign.com - Smiley, finished with some timeouts
* http://halo.ign.com - Smiley, done
* http://hls.gbartone.dev.m.ign.com -> hls.gbartone.dev.m.uk.ign.com, done, smiley
* http://horizonsvault.ign.com - Smiley, done
* http://ie.bestof.ign.com -> uk.bestof.ign.com (localized version of sites?
* http://ie.top100.ign.com - Smiley, (localized version redirected to .uk.)
* http://uk.bestof.ign.com - Smiley grabbing
* http://uk.corp.ign.com - Smiley grabbing
* http://uk.retro.ign.com - Smiley grabbing -> broken by + in url
* http://uk.sports.ign.com - Smiley grabbing -> uk.ign.com
* http://uk.top100.ign.com - Done
* http://iplstore.ign.com - dead, uploaded, Smiley
* http://jloijens.dev.m.ign.com - Smiley, redirected to .uk. version, done
* http://jtai.dev.m.ign.com - Smiley, dead
* http://kaneandlynch.ign.com - Smiley, done
* http://kjaniak.dev.m.ign.com - blocked by auth, Smiley
* http://l2vault.ign.com - Smiley
* http://labs.ign.com - Smiley, done 
* http://lanoire.ign.com - Smiley, done
* http://links.em.ign.com - Smiley, done
* http://littlebigplanet.ign.com - Smiley, done
* http://live.ign.com - Smiley, done
* http://lotrovault.ign.com - Smiley
* http://mag.ign.com - Smiley, done
* http://mail2.ign.com - Smiley, done
* http://mcraft.ign.com - Smiley, done
* http://memoviedia.ign.com - Smiley, dead
* http://mevault.ign.com - Smiley, done
* http://m.ign.com -> m.uk.ign.com, Smiley, done
* http://minecraft.ign.com - Smiley, done
* http://uk.video.ign.com - Smiley grabbing -> broken by redirect
* http://niboppub.ign.com - Smiley, done
* http://nwvault.ign.com - Smiley, done
* http://m.uk.ign.com - Smiley
* http://musichub.ign.com - Smiley, done
* http://mxovault.ign.com - Smiley, done with some timeouts
* http://nchandra.dev.m.uk.ign.com - Smiley, done
* http://o.rpgvaultarchive.ign.com - Smiley, done
* http://potbsvault.ign.com - Smiley, done with some timeouts
* http://rotavault.ign.com - Smiley
* http://rpgvaultarchive.ign.com - Smiley
* http://ryzomvault.ign.com - Smiley, done, some timeouts
* http://sbvault.ign.com - Smiley, dead, done
* http://starcraft2.ign.com - Smiley done
* http://opt-out.emailpreferences.ign.com -> mail.ign.com, Smiley, done
* http://overlord.ign.com - Smiley, done
* http://pawong.dev.www.ign.com - Smiley, auth failed
* http://planetelderscrolls.ign.com - Smiley, done
* http://play.ign.com -> gamespyarcade.com Smiley, done
* http://gamespyarcade.com - Smiley, done
* http://podcast.ign.com -> ign.com/index/podcasts.html, Smiley, done
* http://podcasts.ign.com -> ign.com/index/podcasts.html, Smiley, done
* http://primeblog.ign.com - Smiley, done
* http://promotions.ign.com - Smiley, done
* http://promotools.ign.com - Smiley
* http://publish.ign.com - Smiley, done
* http://rift.ign.com - Smiley, done
* http://rmcadams.dev.m.ca.ign.com - Smiley, auth failed
* http://rmcadams.dev.m.ign.com - Smiley, auth failed
* http://rmcadams.dev.www.ign.com - Smiley, auth failed
* http://rsullivan.dev.m.ca.ign.com - Smiley, auth failed
* http://rsullivan.dev.m.ign.com - Smiley, auth failed
* http://rsullivan.dev.www.ign.com - Smiley, auth failed
* http://share.affiliation.com - Smiley, dead
* http://share.ign.com - Smiley, done
* http://shootmania.ign.com -> ign.com/ipl/shootmania, Smiley
* http://s.insiderdownloads.ign.com -> http://login.ign.com/prime/landing - Need IGN login subscriber, Smiley
* http://skate2.ign.com - Smiley, done
* http://smashbros.ign.com -> supersmashbros.ign.com, Smiley, done
* http://supersmashbros.ign.com - Breaks wget
* http://smcnabb.dev.www.ign.com - Smiley, auth failed
* http://sovault.ign.com -> vault.ign.com, Smiley, done
* http://strangleholdcentral.ign.com - Smiley, done
* http://strategyvault.ign.com -> vault.ign.com, Smiley, done
* http://sts.ign.com - Smiley, done
* http://studio.ign.com - Smiley, done
* http://swgvault.ign.com - Smiley
* http://swvault.ign.com - Smiley, done with errors
* http://tabularasavault.ign.com - Smiley, done
* http://tdu.ign.com - Smiley, done, broken
* http://tford.dev.m.ign.com - not found, Smiley
* http://tford.dev.www.ign.com - not found, Smiley
* http://thhe.ign.com - Smiley, done
* http://ticket.ign.com - Smiley, done
* http://tickets.ign.com - Smiley, done
* http://titanquestvault.ign.com - Smiley, done
* http://tjohnson.dev.uk.ign.com - 404, Smiley
* http://touch.ign.com - Smiley, done
* http://tqvault.ign.com -> titanquestvault.ign.com, Smiley, done
* http://trvault.ign.com -> tabularasavault.ign.com, Smiley, done
* http://twoworldsvault.ign.com - Smiley
* http://uovault.ign.com -> http://vault.ign.com/uovault.html, Smiley, done
* http://vanguardvault.ign.com - Smiley, done
* http://warhammervault.ign.com - Smiley, done
* http://wikihub.stg.www.ign.com - 404
* http://wiki.stg.www.ign.com - 404
* http://wishvault.ign.com - Smiley
* http://witchervault.ign.com - Smiley
* http://www.antis.ign.com -> http://entertainment.ign.com/antis.html 
* http://www.championshipgamingseries.com - Smiley
* http://www.ipl.ign.com -> http://www.ign.com/ipl/
* http://www.kaneandlynch.ign.com -> kaneandlynch.ign.com, Smiley, done
* http://www.mevault.ign.com - Smiley, done
* http://xboxlive.ign.com -> uk.ign.com/xbox-live
=== Ready to grab ===
* http://au.bestof.ign.com
* http://au.retro.ign.com
* http://au.sports.ign.com
* http://wiki.stg.www.ign.com
=== untested ===
* http://au.microsites.ign.com
* http://au.top100.ign.com
* http://au.video.ign.com
* http://crose.dev.m.ca.ign.com
* http://crose.dev.m.ign.com
* http://m.au.ign.com
* http://m.ca.ign.com
* http://m.ie.ign.com
* http://mobile.ign.com
* http://nchandra.dev.m.au.ign.com
* http://nchandra.dev.m.ie.ign.com
* http://nchandra.dev.m.ign.com
* http://nchandra.dev.www.ign.com
* http://o.mobile.ign.com
* http://open.em.ign.com
* http://speedtv.ign.com
* http://sslvpn.ign.com
* http://staging-api.ign.com
* http://store.ign.com
* http://tjohnson.dev.au.ign.com
* http://tjohnson.dev.ca.ign.com
* http://tjohnson.dev.ie.ign.com
* http://tjohnson.dev.www.ign.com
* http://top100.ign.com
* http://v3-api.stg.ie.ign.com
* http://v3-api.stg.m.au.ign.com
* http://v3-api.stg.m.ie.ign.com
* http://v3-api.stg.www.ign.com
* http://vgu.stg.www.ign.com
* http://viashoka.dev.m.ign.com
* http://viashoka.dev.www.ign.com
* http://videopatch1.atio.dev.uk.ign.com
* http://videoplayer.phantom.stg.www.ign.com
* http://video.stg.www.ign.com
* http://vnboards.ign.com
=== These might be asset only hosting sites ===
* http://au.media.ign.com
* http://carsmedia.ign.com
* http://carsmovies.ign.com
* http://codesmedia.ign.com
* http://cubemedia.ign.com
* http://dcmedia.ign.com
* http://entertainmentmedia.ign.com
* http://faqsmedia.ign.com
* http://faqsmovies.ign.com
* http://ffmedia.ign.com 
* http://formenmedia.ign.com
* http://guidesmedia.ign.com
* http://ie.media.ign.com
* http://insdermedia.ign.com 
* http://insiderdownloads.ign.com
* http://insidermedia.ign.com
* http://macmedia.ign.com
* http://macmovies.ign.com
* http://media.ign.com
* http://moviemedia.ign.com
* http://moviesmovies.ign.com
* http://o.faqsmedia.ign.com
* http://o.faqsmovies.ign.com
* http://o.ffmovies.ign.com
* http://o.guidesmedia.ign.com
* http://o.insidermedia.ign.com
* http://o.media.ign.com
* http://o.moviesmovies.ign.com
* http://o.pcmedia.ign.com
* http://o.pocketmedia.ign.com
* http://o.ps2media.ign.com
* http://o.xboxmedia.ign.com
* http://xboxlivemedia.ign.com
* http://xboxlivemovies.ign.com
* http://xboxmedia.ign.com
* http://xboxmovies.ign.com
* http://pcmedia.ign.com
* http://pocketmedia.ign.com
* http://ps2media.ign.com
* http://psxmedia.ign.com 
* http://psxmovies.ign.com 
* http://scifimedia.ign.com
* http://uk.media.ign.com
* http://vaultmedia.ign.com
* http://wiremedia.ign.com
* http://wiremovies.ign.com
* http://c.ffmovies.ign.com
* http://s.faqsmovies.ign.com
* http://s.ffmovies.ign.com
* http://s.moviesmovies.ign.com
* http://s.psxmovies.ign.com
* http://s.xboxmovies.ign.com
=== Redirects ===
* http://cars.ign.com -> http://www.ign.com/
* http://guides.ign.com -> http://www.ign.com/wikis
* http://answers.ign.com -> http://www.ign.com/boards/
* http://ve3dboards.ign.com -> http://www.ign.com/boards/
* http://ffmovies.ign.com -> http://ffmedia.ign.com
* http://ddovault.ign.com -> http://dndvault.ign.com/
* http://bigworldvault.ign.com -> http://vault.ign.com
* http://911.ign.com -> http://tickets.ign-inc.com/
* http://bestofe3.ign.com -> http://games.ign.com/bestofe3.html
* http://dsi.ign.com -> http://ds.ign.com/dsi/
* http://www.ipl.ign.com -> http://www.ign.com/ipl/
* http://xboxlive.ign.com -> http://www.ign.com/xbox-live
* http://download.ign.com - redirect to fileplanet
* http://downloads.ign.com - redirect to fileplanet
* http://dsvault.ign.com - redirect to planetdungeonsiege.com
* http://emailpreferences.ign.com - redirect to mail.ign.com
* http://guidesarchive.ign.com -> http://uk.ign.com/wikis
* http://hjvault.ign.com - redirects to vault.ign.com
* http://ipl.ign.com -> ign.com/ipl
* http://mac.ign.com -> http://uk.ign.com/games/reviews?platformSlug=mac
* http://o.guidesarchive.ign.com -> uk.ign.com/wikis
* http://starcraft.ign.com - Smiley done - redirect to starcraft2.ign.com
== Gamespy Domains ==
=== Ready to grab ===
* http://sslvpn.gamespy.com
=== In Progress ===
* http://lanoirepc.d2gstore.gamespy.com - Smiley, done
* http://gamespyarcade.com - Smiley, done
* http://planetthemovies.gamespy.com - Smiley, done
* http://planetelderscrolls.gamespy.com - Smiley
* http://planetcnc.gamespy.com - grabbed, checking for completeness
* http://planetthesims.gamespy.com - grabbed, checking for completeness
* http://planetfrontlines.gamespy.com - grabbed, checking for completeness
* http://planetcivilization.gamespy.com - grabbed, checking for completeness
* http://planethalflife.gamespy.com - grabbed, checking for completeness
* http://planettransformers.gamespy.com - grabbed, checking for completeness
* http://planetcoh.gamespy.com - grabbed, checking for completeness
* http://planetbattlefield.gamespy.com - grabbed, checking for completeness
* http://planetresidentevil.gamespy.com - grabbed, checking for completeness
* http://planetxmen.gamespy.com - grabbed, checking for completeness
* http://planetquake.gamespy.com - grabbed, checking for completeness, grabbing again.
* http://planetgrandtheftauto.gamespy.com - grabbed, checking for completeness
* http://planettonyhawk.gamespy.com - grabbed, checking for completeness
* http://planetunreal.gamespy.com - grabbed, checking for completeness
* http://planetfallout.gamespy.com - grabbed, checking for completeness
* http://planetageofempires.gamespy.com - grabbed, checking for completeness
* http://planetgearsofwar.gamespy.com - grabbed, checking for completeness
* http://planetcallofduty.gamespy.com - grabbed, checking for completeness
* http://classicgaming.gamespy.com - grabbed, checking for completeness
* http://planetdoom.gamespy.com - grabbed, checking for completeness
* http://planetwwe.gamespy.com - grabbed, checking for completeness
* http://pc.gamespy.com - grabbed, checking for completeness
* http://psp.gamespy.com - grabbed, checking for completeness
* http://ds.gamespy.com - grabbed, checking for completeness
* http://xbox360.gamespy.com - grabbed, checking for completeness
* http://planetfarcry.gamespy.com - grabbed, checking for completeness
* http://planetcrysis.gamespy.com - grabbing omf_
* http://planetmedalofhonor.gamespy.com - grabbing omf_
* http://xbox.gamespy.com - grabbing Smiley
* http://norad.gamespy.com - grabbing Smiley
* http://bf2142portalservices.gamespy.com - grabbing Smiley
* http://arena.gamespy.com - grabbing Smiley
* http://www.gamespy.com - grabbing Smiley
* http://bugsubmit.gamespy.com - grabbing Smiley
* http://ps3.gamespy.com - Smiley, done with 503's at the end.
=== Redirects ===
* http://forumplanet.gamespy.com -> ign.com/boards
* http://forums.gamespy.com -> ign.com/boards/categories/gamespy
* http://planetdeusex.gamespy.com -> gamespy.com (actual site's at planetdeusex.com)
* http://planetelderscrolls.gamespy.com -> planetelderscrolls.ign.com
== 1up.com ==
On 2016-05-24, http://www.1up.com has been thrown into ArchiveBot with .