nyuuzyou/wikis huggingface dataset:
Orain

Orain was a not-for-profit MediaWiki farm. All Orain domains now redirect to Miraheze.
== History ==
Wikis hosted on Orain included All the Tropes, a wiki about storytelling design patterns created in July 2012 when TV Tropes ran into censorship difficulties. On September 16, 2015, a hacker compromised Orain and one or more of its databases, including the All the Tropes database.http://addshore.com/2015/09/downfall-of-orain/https://orainblog.wordpress.com/2015/09/25/orain-compromise-initial-statement/ Fortunately, Orain kept back-ups, but all revisions made after June 2015 were lost. All the Tropes moved to allthetropes.miraheze.org after the incident.http://drunkardswalkforums.yuku.com/reply/132510/All-The-Tropes-Wiki-Project-Part-V#reply-132510
Orain had over 425 wikis.orain.com - list of wikis
== Public backups ==
WikiTeam made backups for ~380 wikis. The dates range from January 2014 to August 2015.orain - dumps There is also an August 2013 package dump of a few wikis.Orain wikifarm dump (August 2013)
== See also ==
* WikiTeam#Wikifarms
==References==
== External links ==
* 
Category:Wikis
Organization for Transformative Works

The Organization for Transformative Works (OTW) is a nonprofit organization, established by fans in 2007, to serve the interests of fans by providing access to and preserving the history of fanworks and fan culture in its myriad forms. Some of their domains are in Alexa's top 1000 and are not fully archived, but would like to be.
* Archive of Our Own (see despite robots.txt, which is restrictive but doesn't literally prohibit crawling)
* http://fanlore.org/ (archived by WikiTeam, a sysadmin will provide newer dumps)
Category:Wikis
Category:Web hosting
Osu!

Osu! is a rhythm game created in 2007 where users download the game itself, create user accounts, download beatmaps, and submit scores.
In the website there are many elements to save:
* Beatmaps
* User accounts
* Leaderboard Scores
= Beatmaps =
Beatmaps can be downloaded as single ".osz" files, which contain all the difficulties.
== Downloading through the website ==
Normally, you'd download a beatmap by going into the website, and (while being logged into an account) searching the beatmap you want (or going to its specific URL) and clicking the "download" button.
This method is also doable in the game itself (in the "osu!lazer" version, which is more cutting edge and contains many more features than the Stable version) via an integrated browser interface of the osu! website.
== Automating the process ==
Beatmaps are numbered alphabetically, starting from 1; however, there are several holes in the numerical lists of beatmaps, so who knows how many have already been lost.
To download a Beatmap manually, you need to make an HTTP request like this:
GET /beatmapsets/<beatmap>/download HTTP/1.1
Host: osu.ppy.sh
Referer: https:/osu.ppy.sh/beatmapsets/<beatmap>
Cookie: <osu session token and other stuff>
Accept: */*
The server should return a 302 Found status code; if it returns something else, something went wrong.
To send requests yourself, you must insert your own osu! cookie in the request; to do so, follow these steps:
# Create an account for osu! (or log in into an existing one);
# Make any request to the osu! website while having the devtools open;
# Copy the "Cookie" header.
Please DO NOT share your Cookie header with anyone alse, as your account will get hacked.
=== 401 Unauthorized ===
As said before, you need to use a valid Cookie header corresponding to your own osu! account; if not, the server will respond with a 401 Unauthorized and a page asking you to log in.
=== 404 Not Found ===
If you get this error code, either the beatmap is lost (unfortunately) or the beatmap doesn't exist (yet).
== Graveyarded beatmaps ==
If a beatmap hasn't been updated for 4 weeks (and is not approved/ranked/qualified/etc.), it automatically enters the graveyard state.
In 2010, Peppy (the creator of osu!) made many old graveyarded maps unavailable to downloadHere's an example.
Fortunately, they are still stored in a separate server (a.k.a archived), and players could ask peppy to make them available again, although today, because the maximum time period peppy was willing to temporainly make those beatmaps available for was 2 years (and 13 years have passed), as far as I know non of teh archived beatmaps are downloadable anymore.
== Throttling ==
I tried running a simple Python script which would download all the beatmaps sequentially, and I started to get ratelimited at about 190 requests. Then, it took around 30 minutes to let me download again. Please note that this ratelimit only affects downloading beatmaps, and not the rest of the Osu! API.
Ownlog.com

ownlog.com (and related fotolog.pl) were one of the oldest blog platforms in Poland. They host about 45 000 individual blogs. Those platforms gradually lose popularity and they seem to be abandoned from their owners (except most critical maintenance). There was high risk of closing this service in a near future as of 2014. Some time in late 2022 the Ownlog website started showing a "Default Web Site Page".https://hackint.logs.kiska.pw/archiveteam-bs/20221012#c329344
== Project status ==
Full item list has been retrieved - pipeline scripts are under development. (No progress on this since 2014.)
PBase

Pbase is a simple photo sharing website which claims to host millions of photos.
Users can have hundreds of galleries with thousands of photos, each with comments and so on.
A forum is also provided, http://form.pbase.com/ , with several hundreds thousands posts and a peak of 1400 users online at the same time in 2016.
Free (trial) accounts are limited to 10 MB in disk space, while paying users can purchase hosting space in increments of 2 GB (https://secure2.pbase.com/pricing.html).
Category:Image hosting
PDF manuals

PDF manuals, guides, data sheets are a valuable resource which is poorly archived and tends to disappear easily because companies die or simply don't care about their out of commerce products.
The main (accessible) resource in Internet Archive is https://archive.org/details/manuals but a series of manual websites have been preserved within web crawls.
== Other archival projects ==
Some examples: .
The web is infested with various domains which crawl manuals and compete for the best positioning in search results for very possible product code. Most of them try to monetize the clicks by obfuscating the PDF or other scam-like techniques.
Manuals can often escape the normal wayback machine crawling because the respective domains require JavaScript searches or even login (for instance on Electrolux websites).
== See also ==
* Datasheets
* FTP
* PDF 2016
* Toshiba Support
== External links ==
* wikipedia:Category:Home appliance manufacturers
* According to a report, the 50 most important home appliance manufacturers are: Amica Wronki (Poland); Arçelik (Turkey); Aucma (China); Bertazzoni (Italy); B/S/H/ (Germany); Candy Hoover (Italy); Chunlan (China); CNA Group (Spain); DACOR (United States); Dongbu Daewoo Electronics (South Korea); Daikin (Japan); De' Longhi (Italy); Electrolux (Sweden); Elica (Italy); Franke Artemis Group (Switzerland); Galanz (China); GE Industrial (United States); Godrej Appliances (India); Gorenje (Slovenia); Gree (China); Haier (China); Hisense Kelon (China); Hitachi (Japan); LG Electronics (South Korea); Liebherr (Switzerland); Mabe (Mexico); Meiling (China); Midea Group (China); Middleby Corporation (United States); Miele (Germany); Mitsubishi Electric (Japan); Nortek (United States); Panasonic (Japan); Rinnai Corporation (Japan); Robam Appliances (China); Saiyi (China); Samsung Electronics (South Korea); Sharp Corporation (Japan); Smeg (Italy); Supor (China); Teka (Spain); Toshiba (Japan); Vatti Corporation (China); Vestel White Goods (Turkey); Videocon Industries (India); Weili (China); Welbilt (United States); Whirlpool (United States); Xinfei Electric (China)
Category:Archive Team
Category:PDF
Pandora.tv

Pandora.tv (판도라TV) was a Korean video sharing website. It shut down on 2023-01-31.
== Shutdown notice ==
[Notice] We would like to express our deepest gratitude to our customers who have supported Pandora TV.
2022.12.13
Pandora TV Co., Ltd.'s video service has put in a lot of effort to provide a better service to many people,but
we regret to inform you of the unfortunate news that the service has been terminated due to various external
environmental changes and deterioration in profitability that has continued for several years.
We would like to express our sincere gratitude to everyone who has loved the Pandora TV video service for a long time.
We will do our best to minimize customer inconvenience until the end of the service.
-Service end date: until 24:00 on January 31, 2023, service access unavailable on February 01, 2023
-All data of customers will be destroyed upon termination of service.
-Video download available date: ~ until 24:00 on January 31, 2023, (limited to downloadable cases)
Parler

Parler was an American social network primarily used by Donald Trump supporters, conspiracy theorists, and far-right extremists. It drew criticism following the storming of the US Capitol on 2021-01-06. In the following days, its app was removed from Google Play Store, Apple threatened and later followed through with removal from their App Store, and Amazon announced that they would suspend Parler's AWS account at 2021-01-10 23:59 PST (2021-01-11 07:59 UTC). Parler had previously said that they would not play a game of provider Whac-A-Mole, which means the site is likely to stay down after that. On 2021-01-10, they stated that they were working on migration but are likely to be down for a while.
On 2021-02-15, Parler came back online after a month of inactivity. Posts made before Parler shut down are not currently avaliable but will be restored.
On 2023-04-15, Parler was shut down after new owner take over.
== Grab ==
The Parler project got a lot of attention. A website shutdown that had made it into the non-specialist news cycle, an archival project that became a political issue, and a one-off AT collaborator's embrace of the celebrity the project gave them combined to - if nothing else - demonstrate that if our reputation is ever at issue, retreating into our shell isn't a completely bad idea. Much of the most negative attention was the result of a "copypasta" whose earliest known appearance was on Facebook, then spread to Reddit, describing a "group of Internet Warriors" who had hacked Parler, created "millions" of administrator accounts, and then downloaded it "for later retrieval by law enforcement, by the public, by Open Source Intelligence communities", primarily so that the first category could go after its users. This attracted a large number of people who either agreed or disagreed with this purported intention.
== External links ==
* API information
* Every Deleted Parler Post, Many With Users' Location Data, Has Been Archived
* A claimed backstory of the "copypasta" by its author
== References ==
Patch.com

Patch.com is a "hyperlocal" news community which is being downsized from its current ~900 sites to ~500.
== Current status ==
In progress. Warrior integration coming soon.
== Patch.com will rate-limit you across all sites ==
Patch.com institutes a rate-limit (some unknown hundreds of requests/hour) across all sites. If you exceed this, all of your requests will be met with HTTP 420s.
If the patch-grab script detects these, it hard-aborts. A kinder solution would be to sleep for some period of time (an hour?) and try again; suggestions appreciated.
Patreon

Patreon is a website and service which allows subscribers ("patrons") to make regular payments to users who offer some ongoing or recurring service in return. Often that means access to digital works hosted and distributed on Patreon's website itself.
== Details ==
The usefulness for archival varies wildly. Content may be:
* hosted on other subscription services,
* hosted publicly (e.g. comics on social media),
* actually hosted mainly elsewhere, e.g. via Dropbox or Google Drive links.
On Patreon itself the files are often protected with time-limited authorisation tokens in a format like:
https://c10.patreonusercontent.com/3/xxxxxxxxxxxxxxxxxxxxxxX1/patreon-media/p/post/11111111/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxX1/1.jpg?token-time=1111111111&token-hash=xxxxxxxxx_xxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxX1%3D
== Tools ==
Patreon helper by dogpixels is a Firefox extension to download the media one has access to.
== See also ==
* wikipedia:Patreon
* Kickstarter
* SubscribeStar
Category:Web hosting
People

Name 	Description 	Website 	Wikipedia
Stewart Brand 	American writer, best known as editor of the Whole Earth Catalog. He founded a number of organizations, including The WELL, the Global Business Network, and the Long Now Foundation. He is the author of several books, most recently Whole Earth Discipline: An Ecopragmatist Manifesto. 	http://sb.longnow.org/ 	Stewart Brand
Michael S. Hart 	American author, best known as the inventor of the e-book and the founder of Project Gutenberg (PG), the first project to make e-books freely available via the Internet. He published e-books years before the Internet existed via the ARPANET, and later on BBS networks and Gopher servers. Hart devoted his life after founding PG in 1971 to digitizing and distributing literature from works in the public domain with free and expired copyrights.	
* http://pglaf.org/hart/
* http://hart.pglaf.org/	Michael S. Hart
Brewster Kahle 	American computer engineer, Internet entrepreneur, internet activist, advocate of universal access to all knowledge, and digital librarian. Kahle founded the Internet Archive and Alexa. In 2012 he was inducted into the Internet Hall of Fame. 	http://brewster.kahle.org/ 	Brewster Kahle
Len Sassaman 	Advocate for privacy, and the maintainer of the Mixmaster anonymous remailer code and operator of the randseed remailer. 	 	Len Sassaman
Richard Stallman	American free software movement activist and programmer. He campaigns for software to be distributed in a manner such that its users receive the freedoms to use, study, distribute and modify that software. Software that ensures these freedoms is termed free software. Stallman launched the GNU Project, founded the Free Software Foundation, developed the GNU Compiler Collection and GNU Emacs, and wrote the GNU General Public License. 	https://stallman.org/ 	Richard Stallman
Marion Stokes 	American access television producer, civil rights demonstrator, activist, librarian, and prolific hoarder and archivist, especially known for single-handedly amassing hundreds of thousands of hours of television news footage spanning 35 years, from 1977 until her death at age 83, at which time she operated nine properties and three storage units. 	https://archive.org/details/marionstokes	Marion Stokes
Aaron Swartz 	American computer programmer, entrepreneur, writer, political organizer, and Internet hacktivist. He was involved in the development of the web feed format RSS and the Markdown publishing format, the organization Creative Commons, and the website framework web.py, and was a co-founder of the social news site Reddit. 	http://www.aaronsw.com/ 	Aaron Swartz
Bela Szilagyi 	American pianist. He amassed a collection of 8,000 VHS tapes with 50,000 boxing fights.
Philosophy

== Philosophy of the Archive Team ==
=== Statement of Philosophy of the Team Archive Wiki ===
This is a quick set of statements regarding why this Wiki exists, and the intentions behind maintaining it, and general philosophies of the owner. It's intended to prevent misunderstandings in the future, and to enable people to focus on adding to the entries without needlessly inefficient battles and in-fighting. Everything's up for discussion, but know who you should be discussing it with. 
==== There's a Lead Guy ====
As it is, I perceive myself (Jason Scott) in the role of "Editor in Chief", that is, the person overseeing the direction and motivations of this WikiThis means it's my fault.. While I am perfectly happy with intra-user discussions about procedure and policy and approach, if things become intractable, parties should all feel they can come to me and make the buck stop somewhere.
The site has no anonymous edits. Come on here and pseudo-nym yourself to your heart's content, but IP-based random hit-and-run leads to a lot of wasted time which we don't have in this time-critical situation. Registering is nearly instantaneous. Do it.
==== Information First, Then Action ====
The best situation is raising awareness of options when taking your data somewhere. To that end, we will maintain pages that are not really related to us archiving anything, but giving users information on tools they can use to archive things. There is a lot of information out there and there is a lot of good work being done, and even a lot of good work being done to categorize this information. Archive Team is just going to be yet another checklist to help you if you come calling.
Similarly, we are likely to end up being a deadpool of sorts, announcing or tracking sites that are failing, on their last legs, or suddenly gone. 
==== What to Save ====
Our priority should be sites where user content was solicited and then provided. If we can, we should try to mirror the whole site.
Next should be sites that are beloved collections of material, or which contain seemingly unique information.
After that, anything goes. It's good to have a backup.
==== Opportunities to make a Difference ====
Wikis can be pretty dreary looking, so if you have skills in the areas of graphic design, mediawiki layout, or just like to keep track of links, please hop on and be a part of it.
This document will be improved.
- Jason Scott
Category:Archive Team
Photobucket

Photobucket is an image hosting service. They were also at one point hosting Twitter images.
== Vital signs ==
Had a major outage though around year turnover from 2019 to 2020.
In May 2023, Photobucket announced to disable all free accounts and their accompanying images.
== Archiving Photobucket ==
PB_Shovel can be used to download all images from an album and all subalbums (recursively).
This version of PB_Shovel has also been modified with a `--links-only` option to crawl a list of every single URLs from the album and subalbums instead of downloading. This makes it possible to use wget to grab all the files yourself, or grab-site to archive with WARC.
Usage: Obtain all the urls of a Photobucket Album, even subalbums (using the -r parameter), and put them in links-<datetime>.txt. This url file can be given to wget or grab-site to download.
 python pb_shovel.py 'http://s160.photobucket.com/user/Spinningfox/library/Internet Fads' -r --links-only
 wget -i links-2016-03-20_02-03-01.txt
 grab-site -i links-2016-03-20_02-03-01.txt
== External links ==
* http://photobucket.com
* https://www.thephotoforum.com/threads/how-to-get-around-photobuckets-watermarks-on-your-images.438126/ - how to modify URLs in order to archive images without watermarks.
== References ==
Category:Image hosting
PhpBB

Mirroring a phpBB forum with wget can be a pain in the ass. Generally mirroring any forum is usually that.
== Recommended Wget switches ==
The following assumes that you are using an up-to-date Wget, 1.14 or later, for WARC support and regex rejects.
Assuming that you want to crawl the forum as a random visitor, not logged in as a user.
To avoid getting a URL with a session ID in it, first grab any page and save the cookie you get:
 wget --spider --keep-session-cookies --save-cookies=COOKIEFILE http://www.example.com/phpBB3/
You now have a cookie, no more session ID URLs, yay! You are ready to mirror now:
 wget -m -np -w 1 -a www.example.com_phpBB3_$(date +%Y%m%d).log -e robots=off -nv --adjust-extension --convert-links --page-requisites --reject-regex='(\?p=|&p=|mode=reply|view=|search.php)' --warc-file=www.example.com_phpBB3_$(date +%Y%m%d) --warc-cdx --keep-session-cookies --load-cookies=COOKIEFILE http://www.example.com/phpBB3/
The reject-regex rules explained:
?p= and &p= only occur when linking to single posts. You are interested in the full threads so these would be redundant.
mode=reply would give you a page where you could enter a reply, you are not logged in so changes are it would just deny that. But even if, there probably would be no meaningful content to save.
view= prevents Wget from downloading view=previous and view=next which are redundant (each thread has a link to the next and previous thread, the resulting URLs have the thread ID of the source thread (wtf...) and the view= parameter). It also prevents downloading view=print which would be redundant again. If you want to extract data later, you might want to include the print view, your choice.
search.php is obviously not wanted.
If the forum blocks random visitors (like you with Wget) from accessing user profile pages, then add mode=viewprofile to the reject-regex!
=== Notes ===
Might want to reject these too:
* mode=post
* mode=email
* mode=quote
* mode=newtopic
* login.php
Do not reject:
keep mode=joined (memberlist!)
== Strategy ==
<none>
Category:Web applications
Piapro

piapro (ピアプロ) is a user-generated content website for the Vocaloid community, operated by Crypton Future Media (the developers of Vocaloid Hatsune Miku).
Users can upload music (オンガク), illustrations (イラスト), text (テキスト, mostly lyrics), and 3D models (3Dモデル).
Media can have a long or short identifier, such as https://piapro.jp/t/-l3D = https://piapro.jp/content/5sd6lhtttbhb170c.
Music, illustrations, text, and Niconico uploads can be linked together. Users can also attach illustrations (either official song art or fanart for a song) to music.
Users could formerly reupload items to the same URL, and old versions were available for enjoying and downloading (https://web.archive.org/web/*/https://piapro.jp/ex_content/*). This feature was deleted at some point in 2023 (judging from Wayback Machine captures); its content is .
POPPRO (https://www.poppro.cn/) is piapro for the Chinese market.
== Downloads ==
Music can be played without logging in to download; however, the served file will be reduced in quality.
 Comparison of https://piapro.jp/t/-l3D = https://piapro.jp/content/5sd6lhtttbhb170c
Streaming 	Download
https://cdn.piapro.jp/mp3_a/5s/5sd6lhtttbhb170c_20121008041228_audition.mp3	https://dl.piapro.jp/audio/5s/5sd6lhtttbhb170c_20121008041228.mp3(query string removed)
5:10 	5:10
128 kbps 	192 kbps
4.73 MB 	7.1 MB
 Comparison of https://piapro.jp/t/GnKT = https://piapro.jp/content/ygqh78yst8sz0mz1
Streaming 	Download
https://cdn.piapro.jp/mp3_a/yg/ygqh78yst8sz0mz1_20091225120919_audition.mp3	https://dl.piapro.jp/audio/yg/ygqh78yst8sz0mz1_20091225120919.mp3(query string removed)
4:01 	3:57
192 kbps 	256 kbps
5.52 MB 	7.24 MB
== See also ==
* Niconico