nyuuzyou/wikis huggingface dataset:
CVE References

The CVE Project has been worried about linkrot in CVE references for quite a while, and now it's time for action! This project started in Tod's junk drawer but it's here, now.
= Reference Links Snapshot =
These are all the CVE reference URLs as of .
* https://transfer.archivete.am/P6uNh/cve-refs.txt
* https://transfer.archivete.am/EZdNi/cve-twitter-refs.txt
Some are useful. Some are not. The ratio is unknown.
= Problem Statement =
Archiving CVE references, ideally, will preserve vulnerability data for future security researchers, both professional and hobbyist, for effectively all time. We're currently in a phase of pretty serious enshittification on the internet right now, so having more than one copy of sometimes key vuln data seems important. Often, CVE ID descriptions are quite short and pithy, with rich vulnerability details stored only in off-site references. These details include:
* Patch availability
* Patch bypasses
* Related CVE IDs
* Precise, line-by-line analyses of vulnerable functions
* Exploitation details, tips, gotchas
* General, related bug-hunting guidance
* Media coverage of affected software
Note that not all vulnerability details are particularly worthwhile; some advisories are just as opaque as CVE descriptions and offer nothing new for the intrepid researcher. Also, exploit details can be particularly touchy -- some people have trouble understanding that full transparency of software vulnerabilities is a net good for civilization. See Tod's blog, The Hidden Harm of Silent Patches for more detail on why this is.
= Goals =
* Archive all CVE references from all time, including the past
** An ArchiveBot job was ran on the URL mentioned above
* Archive "important" CVE references
** KEV seems important
* Archive new CVE references as they come in
** Via the URLs project
* Have some mapping between original and archived references
** It should be easily usable by end users, per CVE ID
** The archive reference should be canonical since it was "true" when referenced.
** The archive should be generated on the spot when CVE IDs are published
* Archives should be distributed
** archive.org seems trustworthy, but it's another egg basket.
** Torrents seem to be a good format for enabling distribution
** Torrents are hard to deeplink into
* Archives should be easily hot-swappable when a particular archive goes sour.
** If archive.org ever does go down, it should be easy to rebuild and redirect
* Maintainers of this whole archive scheme should be easily replaceable
= Issues =
Lots of URLs no longer exist. Some domains in the AB job failed even though they shouldn't have. So once the AB job uploads are done, analysis of the meta WARCs will needed to be done.
== Twitter ==
There is currently no feasible way to archive Twitter threads. At best we can (via socialbot's snscrape support) 3200 recent tweets from a Twitter user.
== archives.neohapsis.com ==
This is a dead domain, so it was ignored. Much of the data has moved to seclists.org though, so maybe the CVE folks need to update the corresponding URLs.
== marc.info ==
This is an alive domain that failed in AB, possibly the bot got banned.
CableAmerica Personal Web Space

Saved by ArchiveBot.
== Vital Signs ==
From ISP Hosting: "treykeyus - The FTP login is broken so it is probably only a matter of time till it dies for good."
== Discovery ==
Sites follow the pattern of http://users.cablemo.net/~USERNAME/.
=== Items ===
* Google scrape
* Bing scrape
* URLTeam scrape
* Wayback scrape
Category:ISP hosting
Cafeblog.hu

Cafeblog.hu is a Hungarian blogging platform.
== Site reconnaissance ==
It went read-only at the end of 2020, saying that it is now a web hosting service, and no more blog posts can be created. Nonetheless, they barely updated the website otherwise to reflect this (like removing Signup buttons and such).
Connections quite often time out.
The operator is the Central Médiacsoport, which, among others, closed EOldal in 2023.
By browsing the different categories, (perhaps) all blogs can be discovered. Such an operation discovered 6,245 blogs, of which 55 are redirecting to other blog hosting providers (i.e. the authors of the blogs moved away).
Therefore, the number of blogs to save is 6,190.
However, there doesn't seem to be a similar index for users. User profiles are linked from blogs as their authors. Commenting users don't have links to their profile pages, if any.
== Archiving ==
user:bzc6p started archiving the website at the end of 2024.
Category:Blogging
Canv.as

Canvas Networks was a website centered on sharing and remixing media, particularly images. The website was founded by the owner of 4chan, Christopher Poole, and backed by Andreessen Horowitz, SV Angel, Lerer Ventures, Founder Collective, and Joshua Schachter. It closed in 2014.https://en.wikipedia.org/wiki/Canvas_Networks
On , a temporary static archive was made available. They expressed great interest in providing a permanent home for Canvas at the Internet Archive.
== Site Structure == 
* uggcf://pnai.nf/fgngvp/nepuvir_guernq_yvaxf.ugzy
* Update : uggc://qenjdhrfg-rkcbeg.f3-jrofvgr-hf-rnfg-1.nznmbanjf.pbz/nyy_hfref.ugzy
== How can I help? ==
The project is not in the warrior yet.
You can run the scripts manually.
New for August 2014!
Start up the warrior and select the Canvas project.
Alternatively for advanced users, you can run the scripts manually. 
== Archives == 
Archives are located at: https://archive.org/details/2014_03_Canvas_Grab
About half of the images were saved. Their server couldn't handle the load until they pulled the plug on .
 archives: archiveteam_canvas
== References ==
== External Links ==
* 
Category:Image hosting
Captain Luffy Forums

== About the Site ==
Viewing some pages requires joining the "18+ group". This will hinder archiving slightly.
== History ==
== Status ==
Archival was requested in #archivebot by akz (who identifies as site moderator geass11), due to an impending shutdown: … "It was stated by the forum owner to me on skype." … "I'll try and get the forum owner to put up the announcement." … "Since he wasn't going to renew his web hosting, which means got a few months at tops." ….
== Sources ==
 * http://archive.fart.website/bin/irclogger_log_a/archivebot?date=2016-01-20,Wed
Channel 9

Channel 9 (2004–2021) was a Microsoft Developer Network-affiliated website for podcasts (incl. video podcasts), interviews, conference talks, etc. relating to software development in the Microsoft ecosystem and unrelated technical material developed for publication by Microsoft developers. In the context of the MSDN shutdown, Microsoft announced on 2021-11-05 that 'most' content uploaded after November 2017 would be migrated to Microsoft Learn. The site shut down later that month between 2021-11-18 21:30 and 2021-11-19 02:50 UTC, though the video files (mostly hosted on sec.ch9.ms) remained accessible at least into January 2022.
Many incoming links were broken with the closure/migration. It's not clear to what extent old content and non-video content will be preserved. Based on the announcement, among older videos, the focus will be on those that are "notable".
== Archival ==
ArchiveBot jobs were run in summer 2020 (restarted after a crash in August) and again in November 2021: . These likely only retrieved English uploads and are possibly incomplete in other regards as well.
User:JustAnotherArchivist retrieved information for all videos from the API in November 2021 as well as the highest-resolution videos based on this list in November and December 2021. More precisely, the largest video file for each API result (as determined by running HEAD requests against each video URL) was retrieved. This data can be found here.
== References ==
Chaos Communications Camp 2011

From their own wiki: The Chaos Communication Camp is an international, five-day open-air event for hackers and associated life-forms. It provides a relaxed atmosphere for free exchange of technical, social, and political ideas. The Camp has everything you need: power, internet, food and fun. Bring your tent and participate!
== Hello, stranger! ==
If you're new to Archive Team, let me quickly recap what we're doing: Archiving. We take historical data and make sure it sticks around for just a bit longer. We've mirrored Geocities before it died, the same with Yahoo! Video, Friendster and many other sites, big or small. Our focus has recently forked into including physical stuff as well, so here's our call for help: 
We'd like you to supply us with historical floppy disks or printed media of any size, with any content, as long as it is in the public interest to preserve it. We'll take good care of your stuff, do our very best to collect as much data and metadata as we can, without destroying any of it.
== A-Team at the CCC Camp ==
The CCC Camp attracts geeks from most of Europe for a five-day collective geek-off. Some of these people are even old enough to have shitloads of archiveables laying around in their basements. Let's use this opportunity to save as much of that as possible!
* For floppies we're gonna need a KryoFlux, currently bargaining at around €100. It lets us dump floppies of any size and format into a preservable data format.
* For physical media like magazines etc., we'll need a scanner. [who has such a device?]
== Dropping the news ==
Once we've settled on a game plan, Jason has said he'd endorse this project, after which we can send a mail out to the CCC mailing list, use their IRC channels, and put a page on the wiki.
== Getting ourselves the needed hardware ==
I, jch, am up for buying a KryoFlux, but unfortunately I don't have the money up-front. I can split with one or more of you and buy you out in September or October. Catch me on IRC if you're interested.
People splitting so far:
* jch
* Nemo
I have a KryoFlux, and I live in Germany (although at the opposite end from where the CCC takes place) and I would be willing to part with my device for the duration of the CCC (or some other well-defined point in time), as long as I will get it back in one piece :-)
I'm in southern Germany, near Stuttgart, so if you're interested, contact me (see my user page for mail address). --Darkstar 22:15, 26 July 2011 (UTC)
== Who's going? ==
* jch
* ersi
== See also ==
* The Camp wiki
Chromebot

chromebot aka. crocoite was an IRC bot parallel to ArchiveBot that uses Google Chrome and thus is able to archive JavaScript-heavy and bottomless websites. On 2021-04-21 the bot was shut down (see #Shutdown), and the captures it made are no longer in the Wayback Machine. WARCs were uploaded twice a day to the chromebot collection on archive.org. For a given item in the collection, you can see what URLs are saved in the warc by looking at the associated jobs.json.gz file.
By default the bot only grabs a single URL. However it supports recursion, which is rather slow, since every single page needs to be loaded and rendered by a browser. A dashboard is available for watching the progress of such jobs.
== Usage ==
crocoite usage documentation
Command 	Description
chromebot: a <url> -r <policy> -j <concurrency>	Archive <url> with <concurrency> processes according to recursion <policy>.
chromebot: s <uuid></code>	Get job status for <uuid>.
chromebot: r <uuid></code>	Revoke or abort running job with <uuid>.
Please note that the commands are case-sensitive.
URL lists can be archived using recursion, for example:
chromebot: a https://transfer.notkiska.pw/inline/UpfR/HollyConrad-tweets -r 1 -j 4
chromebot will assume all lines starting with http(s):// are valid links. Note that the list itself must be returned by the server as an *inline* document, not as a download (attachment).
== Restrictions ==
=== Instagram ===
chromebot has been blacklisted by Instagram. When trying to archive any Instagram.com website, chromebot responds with the following error:
 <Instagram.com URL> cannot be queued: Banned by Instagram
=== Cloudflare DDoS protection ===
chromebot should be able to circumvent Cloudflare's DDoS protection, but scrolling and other behaviour may be disabled after the reload (issue #13 on GitHub).
== People ==
PurpleSym maintains software, scripts, pays the server bills and has administrative access. katocala is a server administrator.
== Shutdown ==
In April 2021, it was discovered that the WARCs written by crocoite had incorrect dates. Namely, the revisit records received the date of the daily deduplication run rather than copying the date of retrieval from the replaced response record, leading to a misrepresentation of when the identical capture was found. Further, all records were presented as HTTP/1.1 with made-up headers, including ones using HTTP/2 or any other protocol supported by Chrome (e.g. WebSockets, HTTP/3). These major data integrity problems led to the bot's WARCs being removed from the Wayback Machine index and the bot being shut down indefinitely. The old revisit records' dates can likely not be fixed reliably because the log information is incomplete, hence a reversal of the WBM exclusion is unlikely.
Category:Bots
Cinema

A film is a series of still or moving images. It is produced by recording photographic images with cameras, or by creating images using animation techniques or visual effects. The process of filmmaking has developed into an art form and industry.
According to The Internet Movie Database there are over 492,000 films and over 600,000 shorts.IMDb Statistics
== Related pages ==
* List of movie databases
== Archive ==
There is a movies collection at Internet Archive.
== Lost films ==
Deutsche Kinemathek estimates that 80–90% of silent films are lost,Why the film archive's own list contains over 3500 lost films.
A study by the Library of Congress states that 75% of all silent films are now lost.Most of America's Silent Films Are Lost Forever
For example, roughly 200 out of over 500 Geroge Méliès' films and 350 out of over 1,000 Alice Guy's films remain in existence today.
* 
* 
* 
* https://www.lost-films.eu/ 
* List of 7000+ Lost American Silent Feature Films
== Public domain works ==
* 
* 
== References ==
Citizendium

The Citizendium is a wiki-based on-line encyclopedia that constantly "pursuits [sic] the highest standards of writing, reliability, and comprehensiveness".
In 2020 after several years of declining activity the remaining handful of users began to discuss shutting down the site. It is unclear when this will take place at the moment, but the site's funds are expected to run out by 2021. There's some efforts to resuscitate the site.
== Dumps ==
* Dumps available (but only the last version of the page, not the whole history): http://en.citizendium.org/wiki/CZ:Downloads
* Using the Wikiteam script, I made a dump of the full histories up to April 2011. They are available through Megaupload and Internet Archive. —Tom Morris 12:07, 18 April 2011 (UTC)
* The latest complete dump was created on 2012-08-09 and is available on the Internet Archive.
== External links ==
* http://citizendium.org
* 
Category:Wikis
Clara.io

Clara.io is a repository of 3D models.
On , a notice was sent to all users by email, stating that the site would be shutting down on .
We are attempting a project for the site. Playback in the Wayback Machine will hopefully work but is as of yet uncertain. As the site doesn't allow explicit "downloads" without an account, someone hoping to reuse the submissions will have to convert them from Clara's apparently proprietary format, which uses a JSON main file as well as several ".bingeom" files that it may refer to.
Club MyCE

Club MyCE was a computer-related discussion forum with a focus on optical media and archival. The forum used the "Discourse" software and was hosted by Communiteq, formerly known as DiscourseHosting.DiscourseHosting - Forum Disabled (2022-07-14)
The forum at club.MyCE.com was unexpectedly shut down without warning in June of 2022. The news website MyCE.com was redirected to MyCE.wiki, even though it is no wiki. What remains is a spammy news website that posts once every few months.
It once again proves what Karl Voit said: Unless a forum is backed up in multiple independent places, it will eventually be memory-holed.Don't Contribute Anything Relevant in Web Forums Like Reddit - Karl Voit, 2020 (Hacker News, presentation, mirror)
== Discussions ==
* 	Hi, what happened to the club.myce.com website? I was very interested in the information on optical discs and for a few months the web has been down
* has myce.com gone down the tubes? - VideoHelp Forum (Thread posted in 2019 during a temporary downtime, but resumed in June 2022.)
* End of Club Myce forum - My thoughts - by former MyCE staff member and news editor.
== References ==
Club Nintendo

Club Nintendo was Nintendo's customer rewards program, in which customers did surveys in exchange for coins, which could be redeemed for rewards (games, playing cards, console accessories, etc.). It shut down on July 30, 2015 in North America and on September 30, 2015 in Europe and Japan.
== Site structure ==
* North America
** https://club.nintendo.com/rewards-details/a/45108.do
*** can also have locale with field ?request_locale= (value can be en, fr)
*** https://club2.nintendo.com/rewards/ssb_poster_big_1.jpg (large preview, there is often more than one of these)
*** https://club2.nintendo.com/rewards/ssb_poster_1.jpg (medium preview, there is often more than one of these)
*** https://club2.nintendo.com/rewards/ssb_poster_thumb_1.jpg (small preview, there is often more than one of these)
** https://club.nintendo.com/rewards-category-details/a/10517.do
*** can also have locale with field ?request_locale= (value can be en, fr)
* Europe
** http://stars.nintendo-europe.com/products/enaumario-hat-198/?locale=en_AU
*** url must have proper locale at the end, there are 16 valid ones (de_AT, de_CH, de_DE, en_AU, en_GB, en_ZA, es_ES, fr_BE, fr_CH, fr_FR, it_CH, it_IT, nl_BE, nl_NL, pt_PT, ru_RU)
*** http://stars.nintendo-europe.com/assets/upload/img/products/912x513/5192-52dc6c6f68048.png (reward image, there is usually 4 per item, 8 if thumbnails are counted)
* Japan
** http://club.nintendo.jp/present/itempop_p201.html
*** http://club.nintendo.jp/images/pop_itemname_p201.gif (item name)
*** http://club.nintendo.jp/images/pop_itemimg_p201.jpg (item picture)
** http://club.nintendo.jp/present/P195/
*** https://club.nintendo.jp/present/P125/pdf/manual.pdf (pages link to some misc files, usually pdfs)
** http://club.nintendo.jp/present/D005/
*** http://club.nintendo.jp/present/D005/manual_K9KJ_00.pdf (pages link to some misc files, usually pdfs)
** http://www.nintendo.co.jp/club/D008/
== Archives ==
Archiving was done through Archivebot:
* stars.nintendo-europe.com
* club.nintendo.jp
* club.nintendo.com
Clutch

Clutch was a video game clip hosting service. It was formerly known as FTW until rebranding to Clutch in February 2018. It shut down on 2020-08-14.
== Shutdown notice ==
On 2020-07-27, the following blog post announcing the shutdown was published:
It is with deep sadness that we are announcing that the Clutch app is shutting down. The app will officially shut down on August 14th. Between now and August 14th you will be able to access and download all your videos. To download your videos you can do so on your profile at Clutch.win
We will be hosting final farewell streams and challenges over the next week. During these events we will be giving away the last of the Clutch T-shirts. We hope that you will come and participate one last time in this amazing community.
Thank you for helping to build this community. It has been a pleasure getting to know you, seeing your clips and sharing in our mutual love of all things gaming. All of us at Clutch will be forever impacted by the many positive experiences we’ve shared with you. Thank you for all you do and we look forward to seeing each of you continue to create amazing content in the gaming community.
If you would like to programmatically access your content feel free to use our API. For example, https://api.clutch.win/v1/users/emilams/posts/compact/ will let you get access to all of your clips. https://api.clutch.win/v1/users/emilams/posts/ will give you more details about each post but require passing cursor as a parameter to page through results.
== Archival ==
Because the website fell over immediately under load, the project was split into two parts, one for the site (clutch) and one for the video files (clutchvideos), both backed by Clutch's API. We managed complete coverage of Clutch, however videos were only retrieved at default resolution (480p for almost all videos), user profiles are only available as an API response but not as a web page. Profiles of users who didn't have at least one video on the platform in August 2020 may be missing.
== References ==
Category:Video hosting
Cobook

Cobook was an online address book. It was acquired by FullContact in January 2014https://www.fullcontact.com/blog/fullcontact-acquires-cobook/. FullContact announced on 2016-06-20https://www.fullcontact.com/?s=cobook that Cobook would be shut down on 2016-08-15https://web.archive.org/web/20161007144313/https://www.fullcontact.com/cobook/.
== Site structure ==
Users can be discovered through the sitemaps.
*https://cobook.co/mashby
**https://s3-eu-west-1.amazonaws.com/img.cobook.co/e35799d340ab15e04183189c75d50750-130.jpg
***also located at http://img.cobook.co/e35799d340ab15e04183189c75d50750-130.jpg
**https://cobook.co/mashby.vcf
== Archives ==
Archive Team Cobook Grab
== References ==
Code hosting services

This is a list of websites that host code repositories and open source projects:
* Apache Software Foundation - 13gb, ivan` grabbed it
* BerliOS
* Bitbucket
* Gna!
* Github - Estimated size 20tb, Wik has backed up 10tb
* Java.net
* Kenai - Is transferring to java.net
* Savannah
* SourceForge (Downloading the files is easy, source code repositories need some tool to discover their type, then grab.)
* Tigris
TODO: Copy the rest of this list onto this page so we can track it.
=== Defunct services ===
* forja.rediris.es, Spanish site with hundreds of projects
* Gitorious, archived, in read-only mode
* Google Code closed early 2016
* http://gforge.org closed 2013-02-01
=== Code hosting archives ===
* Google Code Archive: this archive contains over 1.4 million projects, 1.5 million downloads, and 12.6 million issues.
* Software Heritage
* http://codearchive.org
* IA has a bunch, but not particularly organized
Codearchiver

codearchiver is a Python tool and an IRC bot for archiving source code repositories or, more generally, version control systems. It produces output in the file format native to the VCS, e.g. Git bundles, and stores the necessary metadata that allows for deduplicating archives of related repositories and for restoring repositories.
codearchiver can be used standalone and is available on PyPI. Only the IRC bot's data ends up in the collection on the Internet Archive. However, standalone use may still be useful for people wanting to create their own archives or wanting to create snapshots of private repositories.
Source code can also be saved to the Software Heritage and other archives.
College Confidential

College Confidential is a group of college admissions forums that has existed since 2004, containing a trove of valuable content from teenagers who have later gotten to attend some of the top universities in the United States. It has extremely high name recognition among students who go to the nation's top universities.
It uses a vBulletin-like format. However, it has recently changed its forum search format such that forum pages that go beyond the 250th page are no longer accessible on the forum index (even though they are still searchable through Google and search interface).