nyuuzyou/wikis huggingface dataset:
Docplayer

Docplayer is a document sharing platform. It provides unlimited upload of documents (after registration). The content uploaded ranges from teaching materials through legal papers to advertising ephemera.
The service started sometime in 2015 and is insanely growing: as of June 2016, more than 17 million documents seem to have been uploaded.
The service is available in several countries, with different TLD endings, but with the same layout: Italy, Hungary, Spain, Poland, the Netherlands, France, Turkey, Brasil, just to mention a few.
== Vital signs ==
Seems to be working fine, but as it is a quite new site, and probably home of much copyrighted content, and somehow the general look of it, suggest to the writer of these lines that it is not really trustable. (Maybe just paranoia, but it is one of ArchiveTeam's core concepts, anyway).
Maybe it could be scraped continuously for already uploaded content, in case something happens? Because it's quite an important site.
== Site structure ==
Unfortunately, original documents are protected with reCaptcha, so it will be a tough cookie. Fortunately, the page of the document already contains the "transcript" of the file (as raw text), which is still more than nothing.
Document pages are numbered incrementally, but have the title as suffix: http://docplayer.net/9935872-Assignment-answer-key.html
So documents are not discoverable with brute force, but fortunately, users are, as in that case there is no suffix – and documents are listed on the user's page: e.g. http://docplayer.net/user/10295256/
Interesting that number of users is also high, newest ID is about the same as for documents.
IDs seem to be unique internationally: e.g. http://docplayer.net/user/246538/ and http://docplayer.hu/user/246538/ correspond to the same user. So incremental user discovery is possible from one place, but there is a problem: the file pages can be opened only with the TLD in which country it was uploaded. E.g. the Hungarian http://docplayer.hu/246631-A-kulonbozo-vasuti-halozatok-torzsvonalak-elovarosi-vonalak-mellekvonalak-nagysebessegu-vonalak.html appears only with the .hu TLD; for other TLDs it gives 404. (This also means that opening the corresponding Hungarian user's page works on any TLD, but except for the Hungarian one, the link to her document directs to a 404 page, so it's sort of broken.)
So discovery could be made with search engines (including the site's own one), and that gives valid results, or doing one incremental user discovery and trying the documents for ALL TLDs. (This latter seems to be a bit inefficient, but it covers EVERYTHING.
SUMMARY: Discovery is simple, in exchange for a bit inefficiency probably everything can be found, but without defeating the captcha, only low-quality result is achievable.
Docracy

Docracy is a self described home for contracts and other legal documents that allows users to access various legal documents and sign them for their own use.
== 2018 shutdown't ==
On February 27, 2018, Docracy announced that it would gradually be shut down, with documents no longer being allowed to be sent for signing on April 30, 2018, no new documents after May 1, 2018, and the site finally shutting down on July 1, 2018.https://www.docracy.com/shutdown
As of 2018-11-05, Docracy was still online, and new documents were still being published. No update on the shutdown situation could be found anywhere, and the shutdown notice page was still online without any changes. The red shutdown notice at the top of every page (see screenshot) disappeared sometime between 2018-05-14 and 2018-05-18.
As of 2021-07-13, the site appears to be normally operational, and the shutdown notice page was removed in early 2019.
== References ==
Does Internet Archive Want a Book

The Internet Archive accepts donations of books, especially hardcover books, for the possibility of scanning them in and being made available (in general) at the Open Library site. Several major sources provide donations of books by the thousand to the Archive. However, there is always a search for new unique books not already scanned or prepared to be scanned.
The primary API for dealing with knowing if a given book is scanned or waiting to be scanned can be found at https://archive.org/want/. It returns the status for an ISBN number for a book including "We want it", "We have it", as well as "We have it but we could use a second copy" (as well as others).
* Does the Internet Archive Have My Media?
* Donate Books App for iOS and Android
* Frequently Asked Questions on Donating Physical Media
* Book packing basics
DokuWiki

DokuWiki is a wiki engine. It is often recommended as a more lightweight, easier-to-customize alternative to MediaWiki.
You can use dokuWikiDumper to archive DokuWiki sites and upload them to Internet Archive.
Currently (2023-05-15), hundreds of DokuWikis on this sheet have not been archived yet.
You can check archived DokuWiki items at Internet Archive.
== Archiving tools ==
* dokuwiki-dumper
WikiBot () is an interface to automatically save DokuWiki, PukiWiki and MediaWiki instances.
== Strategy ==
Although ArchiveBot can be used to dump DokuWiki sites, it tends to explode into a large number of URLs due to the media manager and sitemap being repeated on each page with identical content (for instance, https://www.dokuwiki.org/dokuwiki?do=index and https://www.dokuwiki.org/manual?do=index give the same content, but are distinct URLs). This grows further with the number of subpages and files. This can be resolved with one of the following ignores, but the main page has different names on different wikis and needs to be replaced (some wikis use doku.php directly and others don't, but generally only one is needed):
 ^https?://www\.dokuwiki\.org/(?!dokuwiki\?).*[?&]do=(index|media)(&|$)
 ^https?://www\.dokuwiki\.org/doku\.php\?id=(?!dokuwiki&).*&do=(index|media)(&|$)
Additionally, the media manager has separate parameters for the namespace and the displayed URL. This can lead to the whole list of namespaces being saved for each image. For instance, this URL is for the image devel:ideas:graphy-nosidebar-desktop.png, which is in the namespace devel:ideas, with the image and ns parameters matching. But this URL shows the same image, but with the (empty!) plugin:plugin namespace. Note that clicking the link to a namespace on the UI does clear the image, but this is done via javascript; opening the link in a new tab does not do this (and archivebot does not use javascript). The following ignore resolves this (using (?=) to assert that both the image and ns parameters are present, and (?!) to have the regex not ignore URLs where image and ns have matching namespaces). It does not need a specific page name as that is handled by the previous ignore, though the domain does need to be updated.
 ^https?://www\.dokuwiki\.org/(?!.*[?&]image=([^&]+)%3A(?![^&]*%3A).*&ns=\1|.*[?&]ns=([^&]+)&(.*&)?image=\2%3A(?![^&]*%3A))(?=.*[?&]image=[^&])(?=.*[?&]ns=[^&])
It also is not useful to save the upload and search pages in the media manager:
 ^https?://www\.dokuwiki\.org/.*[?&]tab_files=(upload|search)(&|$)
The taskrunner page produces an invisible image that is embedded in every page, with a timestamp in the URL to prevent caching. Its purpose is to update the serverside search index. This generally shouldn't be ignored since it is actually embedded, but if the above ignores were not added and a large number of junk URLs were already saved, then an equal number of taskrunner URLs will also appear and should be ignored.
 ^https?://dokuwiki\.org/lib/exe/taskrunner\.php
== See also ==
* Wikipedia
* WikiTeam
Dopplr

Dopplr was a social network and itinerary planner.
== Shutdown ==
Dear Dopplr user,
As of November 1st, 2013 we will be discontinuing the Dopplr service.
What does this mean for you as a Dopplr user? You will continue to have access to your data until November 1st, 2013, after which the service no longer will be available.
Nokia apologizes for any inconvenience, and we thank you for using Dopplr.
Sincerely,
Nokia
== Data export and Reports ==
The data can be exported for logged in accounts under http://www.dopplr.com/account
Dopplr used to provide annual reports for its users, here's the direct link to one of those: http://reports.dopplr.com/2009/7fb16e238d5659a6.pdf
Trying to brute-force download PDFs with random hashes does not lead to anything (several 100 000 hashes tried, no PDF downloaded): https://github.com/habi/python/blob/master/hasher.py
== Archives ==
Archives include some user reports that were publicly indexed or linked.
* https://archive.org/details/dopplr_fire_grab_1
* https://archive.org/details/dopplr_fire_grab_2
* Wayback Machine listing of user reports
== External Links ==
* 
* Bye Dopplr
Category:Nokia
Dota 2

Dota 2 is an action-based real-time strategy (or ARTS) game published and developed by Valve Corporation. It is the direct successor of the original Defense of the Ancients (or DotA) mod for Warcraft III, with the main developer at the turn of the 2010s (IceFrog) having been brought on to Valve to create a sequel. The game was known to have been functional as early as December 2010, and debuted in a closed beta with the inaugural edition of The International at Gamescom 2011. The closed beta lasted through the rest of 2011, through 2012 and half of 2013, when the game came out of beta in July 2013. It was later ported to the Source 2 engine in September 2015.
The game's in-game digital item economy as well as its large esports scene has resulted in the creation of numerous other sites and other assets related to the game, these include Dota 2 Lounge, joinDOTA, Beyond the Summit, and similar.
== Vital signs ==
Stable, it is arguably one of Valve's main cash cows, aside from the Steam platform itself.
== Developer forums ==
The game has an official development forum and discussion board, "Dota2 Dev". It was used quite heavily in the early part of the game's history, but with the rise of other platforms such as reddit (including the Dota 2 subreddit), activity from both users as well as Valve alike has dwindled in recent years. It would not be unwise to keep an eye on the forums as a similar pattern followed before the sudden demise of the Steam Users' Forums in May 2017.
In response to an overwhelming amount of support on the Dota 2 subreddit to bug-basher Jeff Hill, an experimental GitHub bug tracker was created in June 2022 to manage the influx of quality bug reports. This has cast some doubts about the long-term future of the developer forums that remain unanswered. More recently, Jeff has stated that "GitHub is simply better than the old dev forums in every way" and that the forums "had a problem where it was hard to know what was important and what wasn't - especially if you're a dev who's not a very strong Dota player", with the caveat that "we're learning how to make it work right now".https://www.reddit.com/r/DotA2/comments/vgmr7i/stop_spamming_the_github_bug_tracker_with/id48rma/?context=3
The forums were pulled offline in August 2022 in favor of said tracker which was made official and moved under the ValveSoftware organization.
== References ==
Dota 2 Lounge

Dota 2 Lounge is a site which numerous Dota 2 players list items they want to trade for something else, similar to the likes of Craigslist. It also provides item betting on select pro matches, the latter of which presumably makes up a significant part of its traffic.
== Vital signs ==
Not terribly likely to go away due to it being a household name in the Dota 2 community, unless Valve were to cave to calls from some to crack down on the item betting part of the site (or its CS:GO counterpart). UPDATE: THEY DID: https://store.steampowered.com/news/22883/
As a result of Valve's crackdown, both Lounge sites' betting areas have since shifted to being a general pro match tracker, but without the betting.
== URL Structure ==
Profiles: 64-bit Steam ID used, such as "profile?id=76561198040048374".
Trades: increments by 1 for each new trade made, such as "trade?t=90024860".
Match pages: increments by 1 for each new match posted, such as "match?m=5962".
A theoretical pain point in archiving would be handling comments on the match pages, which are handled through [Disqus]. Otherwise, ArchiveBot should be able to easily chew down on it.
== See also ==
* CSGO Lounge
Category:Video games
Download.com

Download.com is an Internet download directory website launched in 1996 as a part of CNET. Originally, the domain was download.com, which became download.com.com for a while, and is now download.cnet.com. The domain download.com attracted at least 113 million visitors annually by 2008 according to a Compete.com study
== Archival ==
User:Kyndigs Archived all the downloadable software dating from 1995-present, this archive was completed in August 2019, it will be uploaded in batches to archive.org, total archive is around 3.8TB, excludes mobile stuff as that is not downloadable.
* https://archive.org/details/download.cnet.com_1995-1999_20190830000000
* https://archive.org/details/download.cnet.com_2000-2004_20190830000000
* https://archive.org/details/download.cnet.com_2005-2009_20190830000000
* https://archive.org/details/download.cnet.com_2010-2014_20190830000000
* https://archive.org/details/softwaredownload.cnet.com_2015-2019_20190830000000
DrunkenDonkey

DrunkenDonkey or DD was/is an Italian eMule community. The forum is used to share tips and ed2k links. It went through various shutdowns and migrations under various names: DDUniverse, DDUnlimited, perhaps also DigitalReload.
The forum uses phpBB and has at times preserved the user database across migrations.
It was separate from DDGalaxy (which was seemingly the part more involved in producing releases, while DD only hosted links) although possibly related as they fell together in 2007. It was also separate from the more "established"/pseudo-official eMule community/development/support forums such as http://www.emule-mods.it/ http://www.emule.it/ https://emuleitalia.eu/ .
== Timeline ==
* 2007-10: DrunkenDonkey offline, DDGalaxy chiuso
* 2007-10: Alternative a Drunken Donkey
* 2007-10-27: "famoso sito Drunken Donkey che operava nella pubblicazione di link ed2k ed e' stato sollecitato dal suo provider a rimuovere tutti i link ed2k pena l'immediato oscuramento"
* 2007-11: "1 settimana fa doveva aprire http://www.dduniverse.net"
* 2008-01: DigitalReload
* 2009-05: DD Universe – Il forum italiano con links ed2k per Emule
* 2012-02: DDUniverse
* 2012-06: "La guardia di finanza di Brescia, su ordine del Gip, ha disposto il sequestro preventivo della piattaforma DDUniverse"
* 2012-07: @dduniversenews
* 2013-07-19: DDUniverse cambia faccia: ecco DDUnlimited
* 2015-08: @DDUnlimitedNet
* 2019-09: still up (private forum)
Category:Italy
Category:Peer to peer
Drunken Archiving

<arkiver> !a https://ia801004.us.archive.org/9/items/donate/
<ATGoKart> arkiver: Archiving https://ia801004.us.archive.org/9/items/donate/.
<ATGoKart> arkiver: Use !status 9340m7x3ujpwyvttdiefuwe4s for updates, !abort 9340m7x3ujpwyvttdiefuwe4s to abort.
<ATGoKart> arkiver: Your job for https://ia801004.us.archive.org/9/items/donate/ has finished.
<nico_32> arkiver: seriously ?
<arkiver> :D
* joepie91 was trying not to say anything
<arkiver> :P
<arkiver> yeah
<arkiver> I just thought "just in case"...
 +Flashfire !a https://archive.org/
Major Flashfire: Queued https://archive.org/.
Major Flashfire: Use !status elbebtn0hq9kh2cb3s0g7oo5w for updates, !abort elbebtn0hq9kh2cb3s0g7oo5w to abort.
+Flashfire Hi edit
+Flashfire no
+Flashfire abort
+Flashfire abort
+Flashfire avirt
+Flashfire SOME OBE
+Flashfire NOOOO
+Flashfire SVORT
+eientei95 !abort elbebtn0hq9kh2cb3s0g7oo5w
Major eientei95: Initiated abort for https://archive.org/.
+eientei95 gg
+eientei95 !stat
+eientei95 !stats
+eientei95 !status
Major eientei95: Job status: 103191 completed, 3255 aborted, 595 failed, 119 in progress, 2 pending
+Flashfire Drinking bad
+Flashfire take my ops
+Flashfire NOTCOEX
+Flashfire IOS
+Flashfire TSJE OPD
+Flashfire TSKR OPS
+eientei95 Can someone remove drunk Flashfire's +v
# Mode #archivebot [-v Flashfire] by PurpleSym
+eientei95 Thanks
# Mode #archivebot [+b *!~flashfire@216.21.8.63] by PurpleSym
- PurpleSym has kicked Flashfire (Come back when you’re sober.)
Duelyst

An online game. Shut down on . It is unknown if server software has been reverse engineered, however some related sites were scraped using ArchiveBot, including many game assets.
Archived sites:
* https://forums.duelyst.com 
* https://play.duelyst.com 
* https://beta.duelyst.com/
* https://cards.duelyst.com/
* https://support.duelyst.com/
* https://www.counterplaygames.com/
* https://www.kickstarter.com/projects/keithlee/duelyst 
* https://store.steampowered.com/app/291410/Duelyst/
* https://twitter.com/playduelyst/
* https://duelyst.gamepedia.com/
* https://www.reddit.com/r/duelyst/
* https://duelspot.com/
* http://www.duel-cast.com
* https://duelyststats.info/
Category:Online Games
EOldal

eOldal.hu was a Hungarian service for self-creating websites for business, non-profit or personal use. It seems to have been launched in 2007, and shut down on January 31, 2023 as announced in an entry on their official bloghttps://blog.eoldal.hu/cikkek/nyitooldal/a-weboldal-keszitesi-szolgaltatas-megszuntetese.html.
== Shutdown ==
Excerpt from their blog post dated 2022-10-07:
Additionally, a blog post dated 2022-10-17https://blog.eoldal.hu/cikkek/nyitooldal/megszunik-az-eoldal--mit-csinaljak-a-weboldalam-tartalmaval-.html suggest some ways for users to save or migrate their data. These include downloading posts with the "traditional copy/paste method" (sic!); or using the export feature of the administration interface (originally for backup purposes), and then importing it to estranky.sk (apparently a service with the same engine), "although it has a disadvantage that its interface is completely in Slovakian".
== Site reconnaissance ==
Websites of the users can either be assigned a subdomain of eoldal.hu, or can back a – separately sold – .hu domain.
Fortunately, the Top List page seems to serve as a full index of all websites created with eoldal.hu, including those with their own .hu domain. (The pager at the bottom seems to show the first 10 pages, resp. the first 150 websites only, but you can replace the page number parameter in the URL with higher numbers.) As of 2023-01-22, this list enumerates 17,477 websites.
The websites themselves, i.e. the engine behind them, seems to produce a nicely scrapable, structured set of pages.
== Archiving ==
Between 2022-11-27 and 2023-01-22, user:bzc6p managed to archive all websites, except for those that were, for some reason, unavailable (e.g. unresolvable domain names). The total size of the archives is 874 GiB.
WARC archives of websites (categorized to items based on the first character of the website's domain name): https://archive.org/search?query=eoldal_hu_websites&sort=title
Files of 4KB size are actually empty WARCS; they denote that although the given website was listed on eoldal.hu's top list, its domain name failed to resolve.
Archives are currently not browsable in the Wayback Machine (except for Internet Archive's own – probably incomplete – crawls). A Wayback Machine-like browsability feature is planned for archiveteam.hu, that would include the archives of eOldal websites.
== Sister sites ==
There are at least two very similar websites, eStranky.cz and eStranky.sk, with apparently the same engine. It's unknown if they have the same owner at some level, but as of 2022-11-20, the above mentioned websites don't seem to be closing, unlike eOldal.hu.
== References ==
Category:Web hosting
Easel

Easel was an in-browser web design tool that shut down on November 1, 2014.
== Shutdown ==
Easel Is Shutting Down
Today we will begin closing the doors easel.io. Active today, we are no longer accepting new signups. On Oct 31st, easel.io will no longer be available. We will be refunding all paying customers for the previous month of service, and will no longer charge moving forward.
This is a bittersweet day for us. We love Easel, and we use it daily. We are extremely thankful and appreciative of our loyal users. To those of you who continue to depend on easel.io, we are very sorry.
Shutting down easel will allow us to focus on our current efforts.
FAQ
Will I still be charged?
No. We have stopped charging.
When will I get my refund?
Refunds were processed before this was published but it can take several days to appear on your statement. Please contact us if you have not been refunded by next week!
When the last day I can access easel.io
Oct 31st at 11:59pm pacific time.
How do I get my data out?
We’ve made our project export feature available to all users, which means you can download an HTML/CSS version of your work.
What will happen to my data?
The data will no longer be accessible following Oct 31st.
== Site structure ==
*https://www.easel.io/documents/ID (Easel's page viewer, content is embedded as an iframe)
*https://www.easel.io/documents/ID/content (the actual page)
** ID is a 16-character hexadecimal string (e.g. https://www.easel.io/documents/1370177c55b0e923)
=== Sub-domains ===
*blog.easel.io - official blog (saved by ArchiveBot) 
*data.easel.io - where some of the content is stored (some is also on CloudFront)
*email.easel.io - used for links in emails to Easel's users, main site redirects to Mailgun
*help.easel.io - help site (saved by ArchiveBot) 
*www.easel.io - the main site (small amount from search engine crawl saved by ArchiveBot) 
=== Preliminary crawls ===
*bing crawl
*google crawl
*combined crawl
EditThis

EditThis is a wikifarm. According to our estimates, there are over 1,300 wikis. There is a 2014 backup for most of them.
This farm is quite hard to archive, because of
* old software (MediaWiki 1.15) with several weirdnesses, both at application and webserver level (like directory structure, URL rewrites, l10n in MediaWiki namespace);
* slow servers (even after they fixed their robots.txt);
* very strict captcha and throttling (with unhelpful status codes);
* number of wikis taken over by spam since 2012 or earlier.
The owner clearly has not had time to manage it properly for several years now.
Best results to complete a download with launcher.py and dumpgenerator.py have been reached with the following:
* add --exnamespaces=8,9
* use API and a --delay=60, wait 120 s between each wiki.
If you lower the delay too much, or forget to sleep between some kinds of requests, you can easily enter a loop of endless 503 errors and never get out of it (each request, failed or not, counts for the throttle).
== See also ==
* List of wikifarms
== External links ==
* 
Category:Wikis
Egloos

Egloos was a South Korean blog host created in 2003 and announced that it would shut down on .
Its site is mostly HTML, but there is a bit of Javascript interspersed that requires some custom grab logic.
The grab started with one tracker item per user (plus another for themes). Because the long tail made these look implausible to save in time, and because single transient errors (which the site was full of) would lose all the work of one of these, we switched to item-per-URL. Very late in the grab, we discovered that what looked like a capacity issue of the targeted was actually poor international peering on the part of its ISP - a South Korean warrior ran much faster than the rest. This came too late, though, and we saved (pure guessing) ~30-65% of the sites. Media and RSS feeds stayed up, so we got those afterwards, where we knew about them.
Users can up until .
== External links ==
* 
== References ==