nyuuzyou/wikis huggingface dataset:
Project Newsletter

Archiving all the email newsletters.
== Ideas ==
* Have a separate email address for each newsletter.
** Public access to the email addresses will need to be limited to prevent random spam from entering the archives.
*** Spam could also be prevented by maintaining a blacklist. (this also keeps out script kiddies)
** Have a way for people to forward past 'issues' of newsletters. Submitted newsletter issues can be verified using copies from multiple submitters.
* Problem: a lot of newsletters require you to click a confirmation link in an email. Some solutions:
** Make the archiving tools search the email content for confirmation links and follow them.
** Have a password protected page where people can click confirm links.
*** First solution is better, but the second one could be used if the bot fails to properly follow the confirmation link.
* Problem: some newsletters require the initial email to be replied to for confirmation.
** Solution: Make the archiving tools auto-reply to the first email received from a newsletter.
* Download image URLs in each email with wget.
* Upload received emails to IA in ArchiveBot-like GO packs, and separate files by the receiving address.
* Provide an interface for reading archived emails.
* Have a bot crawl the web looking for more newsletters.
** Should it automatically create a new email address and try signing up for newsletters, or should there be some sort of human intervention?
* Problem: if unsubscribe links are made public and followed, the archiving address will be unsubscribed.
** Maybe unsubscribe links could be removed from each email and the unsubscribe link changed to http://example.com or http://archiveteam.org?
*** Upload darkened copies of the original emails to IA along with the modified ones.
* Have a list of newsletters currently being saved to prevent saving multiple copies of the same newsletter.
==Notes==
* http://www.spamdex.org/ has a bunch of spam archives. Maybe we could collaborate OR at least see how they have it implemented in the backend..
Projects

This page should contain, or directly link to, almost all ArchiveTeam archiving endavours, categorized.
* Current projects: currently active, upcoming and recently finished grandiose ArchiveTeam projects. (Extract of the next two categories.)
* Warrior projects: projects that utilize(d) ArchiveTeam's distributed archiving system.
* Manual projects that need(ed) much more effort than just pushing a button.
* Small projects: small-scale website archiving projects usually done by a single individual.
* Early projects: first archiving endavours on the dawn of ArchiveTeam, in a format nobody is apparently able/dare to touch.
(The box on the top counts projects having dedicated wiki pages, those numbers aren't complete and far don't contain all projects mentioned in the sections below.)
If you know of a website in danger, let us know that on IRC. If it's a larger site, please also mention it on the Deathwatch page. And, after a decision is made on IRC, or if it doesn't need a decision, then, to help things kept documented and up to date, you are encouraged to add projects, or modify their status
* in the appropriate section(s),
* on the project's dedicated wiki page (if any),
* on Deathwatch and/or on Alive... OR ARE THEY.
The box on the top is generated automatically from projects' dedicated wiki pages, so shouldn't be touched.
Important: Contents of sections below are embedded from other pages, that is, don't edit the section, nor this page, but use the "Edit this list" link! (That opens the corresponding page for editing, and after editing, you'll be forwarded to the page containing only that list: don't worry, you didn't delete the others.)
= Current projects =
Currently active team projects you can get involved in.
Last updated: (Edit this list)
= Warrior projects =
ArchiveTeam's past, current and future Warrior projects with details, in a table form.
Last updated: (Edit this list)
= Manual projects =
Difficult, discussion-intensive, human-resource-intensive and audit projects.
Last updated: (Edit this list)
= Small projects =
List of smaller website rescuing projects, usually done by single individuals.
Last updated: (Edit this list)
= Early projects =
List of ArchiveTeam's early endavours, for historical interest, not edited.
Last updated: (Edit this list)
Proust

Proust is a web application for sharing and preserving family stories, images, and so forth. The owners of Proust announced they would be closing it on January 31, 2012, but turned around. Some Proust content is available publicly; materials like the Proust blog are also public.
==I'm not dead yet==
All Things Digital reports that Proust is now _not_ shutting down: http://www.webcitation.org/657kPgXOx
The rest of this page will live on as an archive.
==Where do we start?==
Google has indexed some Proust stories, although it's anyone's guess how complete the index is:
http://www.google.com/search?q=site:http://www.proust.com/story
Proust operates an index of public stories at http://www.proust.com/people. The Proust site map at http://www.proust.com/sitemap.xml.gz also can be used to get some person data, although it, like the other two sources mentioned, isn't guaranteed to be complete.
==Current status==
Grabber code is here: https://github.com/ArchiveTeam/proust-pulling
At present, 15,434 users have been identified; some may be duplicates, as Proust permits URLs of the form /story/\d+ and /story/\w+, and it's possible for the two URLs to reference the same story. Oh well.
The grabber code defers download of private stories to some other process, as the content of private stories cannot be accessed anonymously or with just any Proust account.
Psychology Today

Psychology Today is described by Wikipedia as a "media organization with a focus on psychology and human behavior", with a website featuring "therapy and health professionals directories and hundreds of blogs written by a wide variety of psychologists, psychiatrists, social workers, medical doctors, anthropologists, sociologists, and science journalists".Psychology Today - Wikipedia (revision as of 23 December 2022)
The web site is excluded from the Wayback Machine. An archive.today capture shows it was available until at least February 9th, 2014.Psychology Today on archive.today (capture from 25 Jan 1999 09:15:34 UTC, copied on 9 Feb 2014 00:55:32 UTC)
The robots.txt file of the site denotes the operators' disapproval of the Wayback Machine.https://www.psychologytoday.com/robots.txt (contains "User-agent: ia_archiver Disallow: /") (accessed on 2022-12-31) 
Category:Excluded from the Wayback Machine
== References ==
Ptch

Make a ptch (pronounced "pitch") by combining the photos and videos on your phone, or using media from your social networks, like Facebook or Instagram.
== Shutdown Notice ==
Ptch Joins Yahoo!
We launched Ptch just over a year ago. Our passion and our mission was to give you the best way to make and share beautiful movies made from the photos and videos on your phone. Well, someone noticed!
Today, we’re excited to announce that Ptch will be joining Yahoo! As part of the Yahoo team, we’ll be able to focus our efforts and leverage our technology to make Yahoo’s photo and video platforms the best in the world.
As part of this transition, Ptch will shut down on January 2, 2014. Until then, you can download your ptches on ptch.com or save them to your camera roll from the Ptch app on your phone:
On ptch.com, go to your profile page. Click on the title of the ptch and then click on the download button.
In the Ptch app, go to your profile. Tap the share icon under the ptch and then tap “Save to Camera Roll.”
Thank you so much for being a Ptch user and fan. We are grateful for your support and look forward to bringing you more exciting products through the Yahoo platform in the future.
All the best,
Hans and the Ptch Team
== Archives ==
Collection
A list of URLs was generated with the API.
Category:Yahoo!
Category:Image hosting
Puella Magi

The Puella Magi Wiki is a huge fan-made English-language archive of speculah, theories, plot analysis, fanworks, and other items related to the Puella Magi Madoka Magica franchise. This wiki stands out from others for it's trove of intelligent discussion and analysis, along with its importance to Anonymous on 4chan's /a/.
Most of this wiki was built by miscellaneous anons and their discussion threads on 4chan's /a/ board.
==Making Dumps==
We currently use the Wikiteam dumpgenerator python script to create dumps.
Commands used for generating dumps are below:
 python dumpgenerator.py --api=http://wiki.puella-magi.net/api.php --xml --images # Generates full dump using Mediawiki API
When creating dump archives, it is strongly recommended to use 7zip for the highest compression rates.
==XML Full History Dumps==
Daily Dumps: - Full XML backups built and downloadable from this wiki everyday at midnight. (A true step up from Wikichan or the Insurgency Wiki!)
Easily importable XML dumps containing every edit that any anon has ever made on this wiki. 
* wikipuella_maginet-20120317-wikidump.7z - 3.3 MB. 430 MB extracted.
** Internet Archive
** Mediafire
* wikipuella_maginet-20120315-wikidump.7z - 3.29 MB. ~430 MB extracted.
** Internet Archive
** Mediafire
==Image Dumps==
Images are crucial to the completeness of a wiki dump. 
Archivist's Note: I'm still having some trouble automatically saving the images using the Wikiteam script, perhaps due to Cloudflare protection. Hopefully, the site admins might do it for us...
==4chan Threads==
The 4chan threads are the ultimate origin for most of the content on this wiki.
Archivist's Note: I think we should simply wget the threads from foolz, since they only save thumbnails anyway.
* http://wiki.puella-magi.net/Threads
==Misc.==
Other stuff related in some form or manner.
* - Complete Image packs related to Madoka, possibly collected from 4chan. This needs to be put into the Internet Archive.
==Archivists==
* Iruel - The head archivist of the Bibliotheca Anonoma.
Category:Wikis
PukiWiki

PukiWiki is wiki software written in PHP, and is widely used by Japanese wikis. It was forked from YukiWiki.
== Wiki list ==
PukiWiki活用事例
== Wikifarms ==
* https://www.wikihouse.com/
* https://www.wicurio.com/
* https://wiki.fc2.com/ (Login/Captcha required)
* https://wikiwiki.jp/
* https://gamedb.info/
* https://wikiru.jp/
* https://swiki.jp/
== Archiving tools ==
* PukiWiki-Dumper
WikiBot (#wikibot (on hackint)) is an interface to automatically save PukiWiki, DokuWiki and MediaWiki instances.
== See also ==
* WikiTeam
Punchfork

Pinterest acquired Punchfork and was very excited to share the news that the site was to close on March 31, 2013. Archive Team rescued it.
==Announcement==
From :
:January 3rd, 2013
:Dear Punchfork Community,
:Today we are excited to share the news that Pinterest has acquired Punchfork!
:Since launching in January 2011, our mission at Punchfork has been to help home cooks discover new, high quality recipes and share them with family and friends. It is a mission driven by a belief in the ability of web and mobile platforms to inspire our lives offline--at home, in our communities, and for Punchfork, wherever meals are shared.
:To cooking aficionados, Pinterest needs no introduction. It stands as one of the fastest growing online services in history, and millions of people use it to find recipes every day. Pinterest is often described as a platform for inspiration, and we feel this aligns perfectly with the goals of Punchfork. We couldn't be more thrilled to join forces with the Pinterest team in San Francisco.
:Initially, support for Punchfork will continue, but we will soon be retiring the Punchfork site, API and mobile apps. We believe that a unified destination benefits our users in the long run, and the Punchfork team will focus on contributing to Pinterest as the premier platform for discovering and sharing new recipes and other interests on the web.
:Our sincerest thanks to our community members, friends, and everyone involved with Punchfork since its launch.
:See you on Pinterest!
:Jeff MillerCEO, Punchfork
==Archives==
* My Little Punchfork on archive.org (full archives)
* Search for your username to download your files in ZIP format
PureVolume

PureVolume is a music sharing website primarily aimed at independent artists, which was created in 2003 shortly after CNET pulled MP3.com offline after an acquisition from Vivendi. In many ways PureVolume is a precursor to more popular music sharing websites like SoundCloud and Last.fm.
== Shutdown ==
PureVolume initially announced that it would be significantly changing its website on April 30th, 2018, removing user-uploaded music and other content
We regret to inform you that PureVolume.com is shutting down in its current form. If you were storing music here, you will have until April 30th, 2018 to download it. Thanks for all your support and we look forward to sharing with you the new PureVolume.com in the coming months.
But for reasons unknown, they have pushed back the deadline a few times. The last change was to July 30th, but they officially commenced the overhaul on July 14th.
== Archival ==
A decent part of PureVolume was grabbed through ArchiveBot . The job did not finish in time, and it is not known how much content it missed.
The warrior project was based on artists found in a discovery through the player. It archived all artist profiles as well as pages beneath it (e.g. album list). Images occurring anywhere on the pages were archived as well, but user/listener pages and audio was not covered.
Category:Excluded from the Wayback Machine
Pyvideo.org

Pyvideo.org is python-related video indexer, in January 2016 they announced 1, 2 shutdown 'somewhere in 2016'.
Later, after PyCon 2016 the website posted a "thank you" page stating that they're still alive as they've found 20+ contributors to continue support of the website.
The website provides public dataset of all indexed videos at pyvideo-data in JSON format.
Questions to ask a user of a threatened site

This page will hopefully contain a list of questions you can ask a user of a site that is going down, during the planning stage for a large project. These are meant to be mostly nontechnical.
* What's the background to this site being imperiled? (Only if this is pertinent.)
* Does the site do anything if you access too many pages in a short time?
* What about if you use a VPN or TOR?
* Does the site have NSFW or otherwise restricted pages? If so, how do you gain access to them?
* Does the site, or sections of it, slow down or become inaccessible for stretches of time?
Quizilla

Which ArchiveTeam project are you? Quizilla is like a community-generated BuzzFeed but for younger audiences.
It was started in 2002 by Matthew Nielsen and acquired by Viacom in 2006.
== Shutdown ==
Heads Up! On October 1st, Quizilla will officially close up shop. We want to thank you for being a fan of the site. Want more quizzes? Visit us at teennick.com
== How to Help ==
Start up your Warrior and select the Quizilla project.
== Archives ==
Archives are saved in WARC format at archiveteam_quizilla.
== External Links == 
*
Quizlet

Quizlet is a mobile and web-based study application that allows students to study information via learning tools and games. It is currently used by 1-in-2 high school students and 1-in-3 college students in the United States. Quizlet trains students via flashcards and various games and tests. As of April 30, 2018, Quizlet has over 200 million user-generated flashcard sets and more than 30 million active users. It now ranks among the top 50 websites in the U.S.
While there is not any risk of it disappearing soon, it contains a wealth of knowledge and info that AFAIK has no backup. It's always better to be prepared!
== Archival ==
Quizlet ‘sets’ are incremental, with the earliest public set having the id ‘173’ and one of the more recent sets being above ‘300000000’. They do have an open API (see https://quizlet.com/api/2.0/docs) that returns a JSON copy of each set. An example API result can be seen here. Back of the napkin math shows that 300,000,000 public sets would take about 400 GB to store uncompressed.
== Grabbing the Data ==
A grab for the API responses for all sets was ongoing. On September 23, 2018, Quizlet changed the API key signup process, requiring applicants to fill out a form for manual reviewhttps://docs.google.com/forms/d/e/1FAIpQLSc-1GClkKldE1YbHWLm8hao0qzK9bny6seKay_ENnix-gFwlw/viewform. The grab has been put on hold since then.
In December 2018, Quizlet "paused" issuing new API keys.https://quizlet.com/api/2.0/docs
Grabbed data is available in the infobox on the right.
Quorum

__NOTOC__
quo·rum (/ˈkwôrəm/) noun - the minimum number of members of an assembly or society that must be present at any of its meetings to make the proceedings of that meeting valid.
THIS PAGE IS VERY MUCH IN FLUX AND SHOULD NOT BE CONSIDERED IN ANY WAY BINDING OR PERMANENT.
This page is an attempt to whiteboard the idea of creating a quorum mechanism in the Archive Team - the ability to call together a group of members, have multiple members discuss an issue, and then make a decision on that issue. Currently, a lot on Archive Team "just happens" and it generally works (although we're also a butthurt machine, occasionally). For issues that might be globally affecting the operation, or really matter in a permanent way with the team, it would benefit to have a variety of members be able to weigh in before things move in a direction.
==Scope==
Off the top of one's head, these quorums should be:
* A major pain in the ass (30 days plus), so it's not just used for where everyone meets for pizza or what we call a project
* Set up for not being some majority vote of god-knows-who weighing in and deciding things
* Helpful, not a way to turn a decision into a hurricane of turmoil
* A way to show to anyone who asks later, "Why is it this way" - we have a record of discussion and why the choices came down.
Examples of things that might call for a "Quorum" (this process can have a name other than Quorum):
* Whether to move us from one IRC network to another as our main network
* Whether to accept some funding or be called part of some affiliation
* Bury Jason's body at sea or in some sort of air burial thing
== How a Quorum Process is called to order ==
* An issue arises that represents an existential issue (survival of Archive Team in a meaningful way), or touches the actions of nearly all Archive Team members in some way.
== What Happens During the Quorum ==
* Proposals
* Discussions about said proposals
== What a Completed Quorum looks like ==
* The results of the quorum (including discussions for and against) should be left permanently on the Wiki and elsewhere so that later questions about why something happened can be found. (Also, discussed, if the circumstances have changed wildly.)
== References ==
* Robert's Rules of Order
Qwarc

A Python framework written by JAA for quickly crawling sites and saving them to WARC. It does not do any processing itself beyond HTTP requests and responses; all logic must be implemented by the user.
Its source is found here. Ensure you are using the latest commit on the 0.2 branch: the master branch is outdated and uses warcio (which does not fully follow the WARC specification), so it should not be used.
== Requirements ==
qwarc does not work in Python 3.9 and later. qwarc has very specific dependencies, which will be pulled in if you install it with pip. (A venv is probably a good idea.) You also need to manually run pip install async-timeout==3.0.1.
== Writing a spec file ==
qwarc is self-documenting. A given crawl's grab scripts are put into its meta WARC, e.g. in https://archive.org/download/forum.canucks.com_topic_updates_202309/forum.canucks.com-updates-meta.warc.gz. In this way, you can find example grab scripts.
Spec files are written in Python. qwarc will look for subclasses of qwarc.Item (non-recursively). These subclasses should:
* Define a itemType attribute, to allow qwarc to uniquely identify the subclass
* Implement a generate classmethod, which yields the initial set of items for that subclass
** This does not have to be implemented if it doesn't need to queue anything initially, but remember that at least one subclass needs to queue something or the crawl won't do anything. Items must be strings, as they are stored in the SQLite database.
* Implement a process instance method, which processes an item.
When processing an item, you can:
* Access the item value with the itemValue property
* Call self.fetch to fetch a URL
** Set responseHandler to one of the response handlers from qwarc.util, or define your own
** The return value is an aiohttp ClientResponse object
* Call self.add_subitem to add a subitem
== Tips ==
* qwarc adds its own headers, such as a User-Agent, to the request by default. That means if you, for example, set the User-Agent header while calling fetch, your request will have two user agents, which is probably not what you want. This can be fixed by doing something like:
 def __init__(self, *args, **kwargs):
 super().__init__(*args, **kwargs)
 self.headers = []
Alternatively, you can delete or change specific headers instead of removing all of them.
Qwiki

Qwiki was a startup that focused on automated video production.
== Shutdown ==
Qwiki ♥ Yahoo!
IMPORTANT MESSAGE
Hello,
We are writing today with an important update from Qwiki.
As you may remember, our team joined Yahoo! in July 2013. Now, we’re hard at work creating new products for Yahoo! users.
This means the Qwiki app will be retired Nov 1, 2014.
To download content you’ve created with Qwiki, please use the following link, and enter your Qwiki username/password:
Download Content
We look forward to sharing news of our upcoming launches, and thank you for being a part of our community.
All the best,
The Qwiki Team
== Archiving report ==
Downloading content discovered from Google/Bing results: started 2014-09-30, finished 2014-10-03.
Brute force discovery: started early 2014 October, stopped 2014-11-24.
Time was short, discovery inefficient; ArchiveTeam could only do a partial grab.
== Archives ==
https://archive.org/details/archiveteam_qwiki
== References ==
== External Links ==
* 
Category:Yahoo!
Category:Video hosting