nyuuzyou/wikis huggingface dataset:
Comcast Personal Web Pages

Shutting down October 8, 2015.
== Shutdown notice ==
Please Note:
Effective October 8, 2015, the Personal Web Page service (feature of XFINITY Internet) will no longer be available. Please retrieve all content currently saved on your site before October 8, 2015 to avoid permanently losing your files and information. 
Customers who have activated this feature, please check your email for a special offer.
== Discovery ==
Sites follow two patterns:
* http://home.comcast.net/~USERNAME/
* http://USERNAME.home.comcast.net/
* http://BIZNAME.comcastbiz.net
* http://BIZNAME.comcastbiz.com
=== Items ===
* TODO: Scrape Google
* TODO: Scrape Bing
* TODO: Scrape DuckDuckGo
* TODO: Scrape Twitter
* TODO: Scrape Reddit
* TODO: Scrape links from MediaWiki wikis
* TODO: Scrape the Open Directory Project
* TODO: Scrape the Common Crawl Index
* TODO: Scrape the Wayback Machine
* TODO: Scrape URLTeam dumps
* TODO: Scrape a list of subdomains from DNSdumpster.com
Category:ISP hosting
Comments

Site 	Status	Saved?	Number of Accounts	Number of Articles	Number of Comments
BlogSpot comments 	Online	No 	? 	? 	?
DailyMail comments 	Online	No 	? 	? 	?
Disqus 	Online	No 	? 	? 	?
Wordpress.com comments	Online	No 	? 	? 	?
Yahoo! News comments 	Online	No 	? 	? 	?
YouTube comments 	Online	No 	? 	? 	?
Communism

Sites related to Communism.
== Encyclopedias and libraries ==
* Communpedia, communist encyclopedia
* EcuRed, Cuban encyclopedia
* Marxists Internet Archive, library of marxist authors
== Forums ==
* Soviet Empire
== Cinema ==
* cubacine.cult.cu (temp offline?), Cuban cinema (ICAIC productions)
* DEFA Film Library, East Germany cinema database
* Soviet cinema, Soviet Union films, TV and cartoons
== Music ==
* DPRKconcert, DPRK Music Fan, DPRK Music, North Korean music
Communpedia

Communpedia is a communist encyclopedia in several languages.
* Russian (online)
* English (offline)
* Spanish (offline, mirror in comunpedia.wikis.cc)
* Danish (offline)
See dumps at Internet Archive.
== See also ==
* List of wikifarms
== External links ==
* https://wikiapiary.com/wiki/Communpedia_%28ru%29
Category:Communism
Category:Wikis
Computer Chronicles

Computer Chronicles was a television program aired on PBS member stations from 1983 through 2002. In its first years it was hosted by Jim Warren, and for the remainder of its history, was hosted by Stewart Cheifet. Gary Kildall was a co-host from 1983 to 1990. It serves as an excellent time capsule depicting the rise of computing in various industries and in the home, including the meteoric rise of the personal computer at the turn of the century.
== Archival ==
Full episodes of the show are preserved at the Internet Archive and viewable at https://archive.org/details/computerchronicles. YouTuber danooct1 has also reuploaded episodes, usually sourced from IA, on to YouTube at https://www.youtube.com/user/ComputerChroniclesYT.
Cost of archiving a website

Using wget to archive a website is simple and is ideal for small and medium websites. For very large websites, however, you will end up downloading a lot of pages and files that are not needed. Unneeded pages include duplicate pages, search pages, login pages and "edit message" pages. Downloading those unneeded will waste a lot of disk space. In this case, one may have to spend a little time to study the website's organization in order to determine which pages should be downloaded or not.
==Real-time archiving==
Real-time archiving require scripts to poll new content continuously. Writing such scripts may require a lot of time. One may have to parse the HTML to discover pages with new or updated content. The HTML may be invalid. The encoding of the page may also be invalid. One may have to convert the encoding to another encoding before being stored. One has to handle connection errors or missing/invalid pages. If there exists any media such as images, you have to design a good enough data structure or file structure to store them.
This gets even more complicated when the website content is edited often. A data structure must be designed to store different revisions of the content in a space-efficient manner. One has to periodically re-download content and then compare them with prior revisions to see if anything is edited.
Coub

Coub is a video sharing website founded in 2012 that was supposed to have shut down on 2022-04-01. As of September 2023 it remains online but appears lightly maintained, if at all.
== Shutdown announcement ==
On 2022-03-15, a warning banner was added to the website:
Coub will be closed on April 1, 2022. Don’t forget to download all your videos.
This notice later disappeared sometime between 0515 UTC and 0815 UTC on March 18th, and later, the announcement of the shutdown cancellation had appeared. The next announcement had been appearCoub перейдёт к новой команде и продолжит работу:
Coub will continue its work and will be developed by a new team. Details soon.
== Source videos ==
Coub saved full versions of videos in 360p for videos clipped from YouTube, and they remained even after the original video was removed. Sometime in 2023, these source videos were hidden, but the clipped versions still exist, along with the thumbnail of the original.
 
== Archival ==
A little over 77 TiB of content was fetched before and a short time after the shutdown notice disappeared (until approximately April 1, 2022) through the coub-grab project.
== References ==
Countdown

Countdown will close on June 30, 2012.
From Wikipedia:
Countdown was a long-running popular weekly Australian music television show broadcast by the Australian Broadcasting Corporation from 8 November 1974 until 19 July 1987. It was created by Executive Producer Michael Shrimpton, producer/director Robbie Weekes and record producer and music journalist Ian "Molly" Meldrum. Countdown was produced at the studios of the ABC in the Melbourne suburb of Ripponlea.
Countdown was the most popular music program in Australian TV history. It was broadcast nationwide on Australia's government-owned broadcaster, the ABC and commanded a huge and loyal audience. It soon exerted a strong influence on radio programmers because of its audience and the amount of Australian content it featured. For most of the time it was on air, it also gained double exposure throughout the country by screening a new episode each Sunday evening, and then repeating it the following Saturday evening. The majority of performances on the show were lip synched.''
The Goodbye Letter
Dear Countdown Club Members,
It is with great sadness that we let you know the Countdown website will be permanently closed down at the end of June. The associated Countdown Twitter and Facebook accounts will also be closed at the same time.
As the final curtain comes down, we wanted to thank you all for re-living the Countdown memories with us over the past few years. It's been a blast.
With one final bow,
== How to help archiving ==
=== Notes ===
== Uploading your data ==
=== Archive status ===
== Site structure ==
== Tools/Archiving ==
Course Hero

Course Hero is a website that hosts academic resources. The resources are downloadable for a subscription of $119.40 per year (automatically renewed). "With a Premier membership, you will get 30 unlocks per month starting on your subscription date. Your unlocks will refresh every 30 days, unused unlocks don’t carry over. If you receive unlocks from uploading documents, the unlocks expire after 30 days from receiving them." Or: "Users can upload 10 documents for 2 unlocks, 30 docs for 10 unlocks, 50 docs for 20 unlocks, and every 10 docs after 50 will receive 4 unlocks." (according to http://support.coursehero.com/hc/en-us/articles/200351044-Do-I-have-to-pay-How-much-does-it-cost- ) (An "unlock" is full access to a document.) Has several million documents, as well as flash cards and other resources. It's sort of a piracy pyramid scheme for cheaters whereby users are coerced into uploading their professors' content in exchange for material other users have uploaded, or else paying for it, at least as far as I can tell. Sleazy business model aside, it should be archived.
The only ways to archive it, as far as I see, would probably be: finding a way to get to their site's contents without a subscription, getting them to donate their documents to IA (probably unlikely, but maybe they'd do it if the items were darked), or using a bot to bulk upload documents to get access to the existing ones (assuming 7 million existing documents, that would require uploading 17,500,000 documents). The third option would probably take lots of email addresses, Course Hero free accounts, and IP addresses.
== References ==
== External links ==
Category:Image hosting
Coursera

Coursera is a company that offers MOOCs (Massive Open Online Courses) for free.
In 2014, Coursera introduced a new platform and moved all courses there, except for a few hundred ones. On June 30, 2016 access to the old platform ended, which resulted in the removal of 472 courses.http://coursera.tumblr.com/post/145882467032/courseras-transition-to-a-new-technology-platform Guides were distributed to save these old courses.http://makemeflow.org/advice/2016/06/how-to-download-courseras-courses-before-theyre-gone-forever/
Someone created a Google Docs spreadsheet with a list of the courses on the old stack and links which were useful.https://docs.google.com/spreadsheets/d/1kaWxZG3krI83WfdzlExWL-_S5UBai-CYuofGR5DZcd0/edit?pref=2&pli=1#gid=1433118913
== Announcement ==
:"Maintaining two platforms requires significant resources, and in order to focus all of our efforts on further improving the new platform experience, we decided last year to move all active courses to the new platform. Over the past several months, we have worked closely with our partners to transition their courses to the new platform. Last week, we emailed learners who have taken courses on our old platform to notify them that access to the old platform will end on June 30, 2016.
:There are a few dozen courses on the old platform that will not migrate to the new platform, and thus will not be available after June 30th. These include courses that are out of date (e.g., medicine and technology courses that do not reflect recent research and development breakthroughs), courses that have been updated and relaunched under another title on the new platform, and a few courses that our university partners have chosen to discontinue for other reasons."
== Archives ==
Archive Team was able to download the endangered courses, which weigh 1,2 terabytes in total.
The content has been packed up into WARC files and can be found in the archiveteam_coursera collection, and URLs are directly available in the Wayback Machine too.
Some courses, like Intermediate Organic Chemistry and Epigenetics are not on this list.
== References ==
CyberCiti

CyberCiti is a popular Linux-related blog that is at least a decade oldComments from 2010, which contains comprehensive guides on a wide variety of commands and software.
CyberCiti is owned by Vivek Gite, who used to operate a Linux help forum, "nixcraft.com". In the year 2015, Vivek shut down that forum without a warning, leaving behind the following love letter to its contributors and visitors:cat nixcraft.com >/dev/null
Notice the lack of remorse over deleting years of volunteer contributions, followed by his shameless advertisement plug of his blog and social media profile.
Sites owned by Vivek Gite may be helpful in the short term, but are not reliable as a long-term reference. If there is anything of value, save it locally.
== Vital signs ==
Its exclusion from the Wayback Machine puts the entire site history at risk should it shut down in future. Deletion of articles likely causes unrecoverable dead links from other sites such as internet forums.
This should be kept in mind when linking to guides on CyberCiti. If possible, search an alternative or create a guide on an archive-friendly site like Wikiversity.
== References ==
Category:Excluded from the Wayback Machine
DNAinfo

DNAinfo was a local news venue from New York City/Chicago which suddenly shut down on 2017-11-02. Gothamist, DCist, LAist, Chicagoist, SFist, and Shanghaiist were also affected by this shutdown. The websites came back online the next day.
All seven sites were archived through ArchiveBot.
An archival copy is expected to be provided to ArchiveTeam voluntarily by the website administrators.
(Londonist and Torontoist, originally also part of this network, were transferred to other companies in 2009/10 and thus unaffected by the 2017 incident.)
Category:News
DNS History

DNS History is a DNS record archive provided by .
It was scheduled to close on due to "funding issues". 
== Announcement ==
According to a dated :
:"dnshistory.org will be closing down in a couple of weeks - it has been a fun project over the years thanks folks."
Also, on the , on the right:
:"This site will shut down on the 10th July 2016 due to funding issues."
== Final announcement ==
On the main page, appeared on :
:"DNS History has now shutdown, any updates are for my personal requirements only - the site may go up/down as I need the resources it uses. The number of servers dedicated to the DNS History project is now ZERO. Due to the ArchiveTeams self-righteous attitude CloudFlares DDoS protection has been enabled. They are quite open about their attitude of ignoring robots.txt files and work around blocks on their user agent - this as abuse."
== Discovery results ==
Out of the 1365 TLDs,
* 1114 do have pages
** 1110 are 
** .info: 251346 pages
** .org: 288405 pages
** .net: 422569 pages
** .com: unknown (expected to be millions of pages)
* 251 don't have a single domain: 
== Archiving ==
After ArchiveTeam started the project on , dnshistory.org staff shortly activated CloudFlare in order to block our access to the site.
bzc6p contacted them asking for cooperation, but they that "there is no way you can archive the >billion pages on the site and trying to do so is causing issues for other users on the site."
Later CloudFlare seemed to have been deactivated, but the site was broken for a month, giving 500 errors. However, as of , the site is up, with various queries, including search, still functioning, with the closure notice still up.
On , ArchiveTeam restarted its efforts. On , although the site seemed to deal with the load, and the site has officially been closed for 42 days, they started to ban our crawlers. On , they deployed their impenetrable Cloudflare browser checker again.
On , a final shutdown notice appeared (see above).
ArchiveTeam was hardly able to save anything.
Since at least 2018, the site has had numerous prolonged downtimes returning blank HTTP 500 responses on all pages except the homepage.
DOS Floppies

This page shows how to dump and image 3.5" and 5.25" MS-DOS/Windows floppies.
The examples all use dd and ddrescue, and were tested on Linux machines only.
Please note that due to the nature of the medium, successive reads from a damaged area may exacerbate the damage. 
(All commands were done on a computer with a regular 3.5" floppy drive hooked up thru IDE, not a USB drive).
At the most basic level, to view the files, and do basic operations on a disk, try mtools. Here is documentation on using it- the page is old, but it contains the essentials about mtools.
Basic image using DD: 
 dd if=/dev/fd0 of=AOL25-test.img conv=notrunc 
 The most basic attempt at making an image- it stops when it hits an error.
Imaging a disk with one or more errors on it:
 ddrescue -d -n -b512 /dev/fd0 Lemmings.img Lemmings1 
 This attempts to image a floppy, continuing when it hits an error, and leaving notes in a log file where the gaps are. 
Using split functionality to recover additional data from disk:
 ddrescue -d -b512 /dev/fd0 Odell-Down-Under-disk2-copy.img Odell2 
 This allows ddrescue to split the damaged areas, and then tries to recover data from each new range.
Telling ddrescue to try again from scratch:
 ddrescue -d --retrim -b512 /dev/fd0 Sea-Rogue-disk3.img Sea-Rogue3 
A very powerful ddrescue imaging attempt:
 ddrescue -d --retrim -r 3 -b512 /dev/fd0 Sea-Rogue-disk3.img Sea-Rogue3 
 Try from scratch again, reading in reverse this time, with 3 tries for each error.
Category:Physical media
DPReview

DPReview (Digital Photography Review) is a website and community on digital photography, founded in 1998.
On 2023-03-21, it was announced that DPReview would shut down on 2023-04-10. Shortly before the deadline, on 2023-04-07, and after receiving a lot of feedback from disgruntled users, they rescinded the shutdown, announced that the site would 'remain available as an archive' in the future and would stay operational while the archival work was going on.
ArchiveBot jobs for the forums () and the main site () were launched, but both got blocked. In any case, these jobs were unlikely to finish in a timely manner.
A DPoS project began in early April.
On 2023-06-20, it was announced that had acquired DPReview and that "the site will continue to operate as it was before, with all editorial coverage and site features remaining the same, and all historical content accessible."
== References ==
DailyBooth

Closed on December 31st, 2012: http://thenextweb.com/insider/2012/11/12/is-original-photo-sharing-service-dailybooth-closing-down/
We've saved everything some days before the deadline, for a total of about 10,055 GiB of material.
The archives are now on archive.org. Use this handy search page to find a particular username.
Category:Image hosting
Dailymotion

Dailymotion is a video hosting and -sharing platform, similar to YouTube but based in Paris, France.
Since their website layout was redesigned, the website heavily relies on JavaScript and AJAX/XHR (also known as “extraload” or “extraloading” for YouTube comments), which makes it inaccessible to the Wayback Machine.
Dailymotion's comment feature was removed in 2018Reddit user: “Is it possible to comment on videos anymore?”, making all existing comments invisible. As of May 2019, there has not been any official comment about this removal.
Dailymotion has a sorted index of every video that is available on the platform, which has been at last archived using ArchiveBot on May 10th 2019 ().
In 2013, Dailymotion came the closest to its demise due to nearly having been purchased by Yahoo, which would likely have lead to its shutdown regarding the history of Yahoo. However, the acquisition was blocked by the French government.France blocks Yahoo bid for video site
==Inactive Videos==
On , Dailymotion sent an Email to users informing them that videos that have not been accessed within the last 12 months will be deleted 3 months after the Email notification.https://old.reddit.com/r/Archiveteam/comments/1idg2nh/dailymotion_start_deleting_inactive_videos/ A DPoS project is planned to save them.
Category:Video hosting
== Tools ==
* dailymotion-dl: hacky shell script
== References ==
DataRefuge

Data Refuge (#datarefuge) is a project concerned with saving (United States) federal climate and environmental data, in response to the 2016 US Presidential election. It's an experiment by the Penn Program in the Environmental Humanities (ppehlab.org/). They collaborate with Project_ARCC and University of Michigan Libraries, among others.
They held real life DataRescue events and can be followed on their Twitter account. 
== Tools ==
Their event toolkit can be found here. Instructions can be found here. They maintain a list of gov. climate datasets here.
== Results ==
They form part of the University of North Texas Libraries' End of Term Presidential Harvest 2016, in collaboration with Library of Congress and the Internet Archive. Data they grab may also be hosted by either the organisations holding the Data Rescue events and/or in a custom repository built using Amazon Web Services integrated with CKAN.
Daum Agora

Daum Agora was a forum where users could post their opinions on politics and society or submit a petition. Posts to the Agora has historically caused debates about freedom of the press in South Korea with prosecution over a post on Agora.
On December 3, 2018, Kakao, the operator of Daum, announced that they would disable new posts and comments by January 7, 2019. The backup feature would allow the post author to back up their own posts, but their comments would be lost in the backup. Their annoucement noted they would purge the data by April 1, 2019.
The forum was archived through ArchiveBot .
Daum Tvpot

Daum Tvpot (다음 TV팟) was a South Korean video website owned by Kakao Corporation, which had (effectively) acquired a Daum portal site which originally had the video website. It consisted of multiple sections, of which a major portion was a public video hosting at http://tvpot.daum.net/v/<videoid>.
On December 19, 2016, Kakao announced that Tvpot will merge into Kakao TV (Korean), and the public video hosting would cease by June 30, 2017. It appears that it was still possible to transfer your uploaded videos to Kakao TV until 2018-07-11http://tvpot.daum.net/, 2018-11-05. The corporation itself only allowed the backup of videos you have uploaded; if you didn't or couldn't request the backup for some reason, it's gone forever. As it was one of the major video hosting website in Korea this represents a major blunder for users.
== Technical Information ==
* Sequential ID assigned to both public and non-public videos (it is also used for hosting videos for the portal)
* Max video ID is somewhere around 82,000,000
** In the initial sampling by User:lifthrasiir public videos takes about 10--20% of all video IDs, which translates to 10--15M public videos
* The links to available videos per resolution are located at http://kakaotv.daum.net/api/v1/ft/cliplinks/<videoid>/raw, but it would require the session ID from the video page
* The actual video is located at tvpot.cdn.videofarm.daum.net
== References ==
Category:Korean
Degree Confluence Project

The Degree Confluence Project aims to visit and collect geographically representative photographs and information for every "confluence point" (integral latitude and longitude degree intersection) on Earth.
Stats: 12798 contributors have submitted 110656 images covering 6349 "primary" confluences, 604 "secondary confluences" (e.g., points in the water with no land visible), and 374 incomplete confluences.
==Vital signs==
Stable, but with no known complete mirrors.
Category:Image hosting
DemoDrop

DemoDrop was a platform where artists could share demo tracks, with the intent of being noticed by record labels, DJs, producers, etc. It shut down on 2021-10-30, prior to the announced deadline of 2021-11-01.
There was an attempt at archiving DemoDrop with ArchiveBot, however extreme rate limits immediately brought this to a halt.
User:JustAnotherArchivist archived all track and most user metadata through the site's API as well as about 83 % of the track streams. The stream download was on track for finishing in time when the site shut down early.
== References ==