nyuuzyou/wikis huggingface dataset:
Google Groups Files

Google is challenging AT again...
This notice appears on Google Groups pages:
----
Zipped versions of the pages and files associated with this group will be available for download until August 31, 2011. After this date, this feature and the zip file downloads will be turned off permanently.
----
A script is available that searches Google Groups directories and downloads the ZIP files of individual groups. The script uses a Google App Engine hosted app for coordination.
== Status ==
Uploaded to The Archive Team Google Groups Collection.
We have downloaded everything. Please contact SketchCow for an upload slot.
2011-08-27:
directories: 577199
groups: 1526946
2011-06-04:
directories: new: 59926, done: 25585
groups: new 552075, done: 70804
2011-06-14:
directories: done: 76080
groups: done: 163749
2011-06-16:
directories: NEW: 80942, PROCESSING: 7, DONE_DIR: 81644
groups: NEW: 880230, PROCESSING: 97, ERROR: 5825, ADULT: 3893, DONE_GRP: 172148
2011-06-20:
directories: NEW: 71364, PROCESSING: 6, DONE_DIR: 101141
groups: NEW: 778080, PROCESSING: 290, ERROR: 10249, ADULT: 4177, DONE_GRP: 298122
completion rate: directories: 170/hr, groups: 2213/hr
2011-06-28:
directories: TOTAL: 243898, NEW: 105872, PROCESSING: 15, DONE_DIR: 138011
groups: TOTAL: 1245968, NEW: 767342, PROCESSING: 44, ERROR: 10944, ADULT: 4236, DONE_GRP: 463402
completion rate: directories: 337/hr, groups: 865/hr
2011-07-19
directories: TOTAL: 443013, NEW: 159064, DONE: 270836
groups: TOTAL: 1505795, NEW: 558656, DONE + ERROR + ADULT: 947139
completion rate: directories: 231/hr, groups: 1727/hr
new discovery rate: dirs: 0/hr, grps: 2/hr
== Script ==
=== Requirements ===
(ba)sh, wget, grep, curl
=== Usage ===
* Normal operation
./ggroups_zipdl.sh
* Discover only (no downloads to store)
./ggroups_zipdl.sh discover
* Download only (no discovery of new groups)
./ggroups_zipdl.sh download
=== Issues ===
-
Category:Google
Google Helpouts

Google Helpouts is a service that lets users share their expertise using technologies from Google Hangouts. Shutting down on April 20, 2015
== Site structure ==
*https://helpouts.google.com/104794915167914988036 (profile)
**https://helpouts.google.com/104794915167914988036/ls/9cbe84c41b20f14b (helpout)
**https://helpouts.google.com/api/offer/104794915167914988036/9cbe84c41b20f14b?format=JSPB&authuser&xsrf=null&nr=2174&mr=1424403240716&_reqid=48096&rt=j (helpout comments)
*** introduction video is hosted on YouTube (doesn't appear to be public)
*https://helpouts.google.com/search?cat=/t/c/01&num=10 (search page)
** no apparent limit on the num field
** number at the end of the cat field is sequential from 00-10 (with a couple gaps)
*** cat field also accepts /t/c or /t, which appear to return results from all categories
** there's also an optional sort field, it's value can be either r, p, or t.
== Archives ==
Archiveteam Google Helpouts: Helping Hand
Category:Google
Google Maps

Google Maps is a digital world map captured and served by Google.
== Information ==
Historical versions of Google Maps satellite images and StreetView imagery are currently accessible through the Google Earth software.
== Metadata ==
Google Maps contains a lot of metadata about physical locations that would be useful to archive, including organisations and their websites and when the organisations are permanently closed. This could be useful to discover websites that are likely to shut down soon. DPoS could be used to save this data and send outlinks to ArchiveBot, Y and or URLs. jo72/brutecat has already archived 300m+ Google Map location IDs and metadata, as well as ~2 billion Google Maps reviews, uploaded the data and released the code.
Google Moderator

== Shutdown announcement ==
Unfortunately, Google Moderator has not had the usage we had hoped, so we’ve made the difficult decision to close down the product. We want to ensure users have enough time to export their Moderator series data using our Takeout tool. Please take a look at the timeline below for more information.
March 30, 2015 is the first day that you can download your Google Moderator data from Takeout. Your data from past Moderator series will be available in Takeout for at least two years.
June 30, 2015 will be the last day you can create a new series, ask a question, or vote on a question. For the month of July, Google Moderator will be “read-only.”
July 31, 2015 is Moderator’s last day. The site will no longer be available in any form, but you will be able to access data from past Moderator series through our Takeout tool for at least two years.
Thank you for asking and voting on questions with Moderator over the past several years.
== Site structure ==
*https://www.google.com/moderator/#N/e=ID
** N is a one or two digit number (highest i've seen is 33, but they could go higher), ID is the item id
*** (example: https://www.google.com/moderator/#15/e=20fdbf)
*** Highest discovered so far is 21fdbf
** https://www.google.com/moderator/#15/e=20fdbf&t=20fdbf.40 (subpage)
*https://www.google.com/moderator/g/csv/moderator-ID.zip (item in csv format)
** (example: https://www.google.com/moderator/g/csv/moderator-20fdbf.zip)
== Archives ==
https://archive.org/details/archiveteam_googlemoderator_20150929152735
Category:Google
Google Plus Comments on Blogspot

== History ==
As part of Google's efforts to integrate Google+ with a variety of its services, a feature was made available to allow blogs hosted on Google's blogging platform known as Blogger/Blogspot to use a Google+-based commenting system rather than the previous commenting system that offered a variety of authentication methods. Google+-based comments were never made a mandatory feature, but many blogs opted-in to using them.
== Shutdown Information ==
As part of the Google+ shutdown, Google announced on 2019-01-30 that Google+ comments would be removed from Blogger blogs on 2019-02-04. Blogs were encouraged to revert back to the previous commenting system.
Beginning on 2019-02-04 and through the following several weeks, Google+ comments were gradually removed from Blogger blogs, however, because the Google+ comments were implemented as an embedded comments widget, the data remained accessible via a simple URL until the Google+ API shutdown on 2019-03-07. The API shutdown occurred between approximately 18:27 UTC and 20:27 UTC on 2019-03-07.
== Archive Information ==
Contributors tech234a, afrmtbl, jopik, and glmdgrielson from the Internet Trash Heap Discord server worked together to lead the archiving project for Google+ comments on Blogspot blogs. Comments were archived between 2019-02-26 and 2019-03-07. Blog URLs were discovered using the Common Crawl's CDX API and the Internet Archive's Wayback Machine CDX API. In some cases, legacy comments that appeared in Google+ comment widgets were also archived. Slightly less than 30 million blog domains were scanned. An archive was posted in JSON format on the Internet Archive with data from approximately 181,565 blogs that had Google+ comments associated with them. In addition, a searchable archive of this data was created and posted to the Internet Archive. The JSON schema is available on the archive's Internet Archive page.
== Future ==
A possible future project would be to allow users to re-import Google+ comments for inclusion with legacy comments on their blogs. This would be possible by exporting the blog as XML, adding the additional comments to the XML file, and re-importing the blog's XML file. This can be done manually or via one of Google's OAuth APIs.
In November 2023, a list of 34,285,898 blogs (filtered to 34,234,015 items) discovered during this project, as well as a list of containing mappings between 64,921 custom domains and their associated Blogspot subdomains, also collected during this project, were submitted to Archive Team's general Blogger project.
Google Poly

Google Poly was a platform for 3D objects. It became read-only after 2021-04-30 and shut down on 2021-06-30.
== Downloading Assets ==
=== Manual ===
The Poly web interface appears at https://poly.google.com/view/<id-of-item>. The web interface downloads sketch.gltf and sketch.bin. Links to sketch.gltf, sketch.bin and sketch.tilt can be found in the html page in the object passed to the second call of AF_initDataCallback. This works for models with any license.
Assets IDs can be grabbed from search using https://poly.google.com/search/<search-term>. Search for links to ./view/<some-id>. Unknown how to get past the first page.
=== API ===
The Poly API requires an API key. Each API key has a limit of 3000 queries per minute, and no limit per day/monthhttps://developers.google.com/poly/develop/api#usage_and_quota.
It's possible to get all CC-By assets using https://poly.googleapis.com/v1/assets. This returns an array that contains the information for each asset. The formats object contains links to the 3D model files, (sketch.gltf, sketch.bin, sketch.tilt and thumbnail.png) and these can be downloaded without an API key.
Other assets IDs would have to be scraped from search using https://poly.google.com/search/<search-term>. For each asset, the ID from the URL can be used to retrieve the same info as above from https://poly.googleapis.com/v1/{name=assets/*}
== References ==
Google Reader

Google Reader was an RSS feed reader, launched by Google in 2005 and killed off in 2013.
== Shutdown notification ==
On the March 13, Google announced that they'll "spring clean" Google Reader at Official Google Reader Blog:
:Powering Down Google Reader
:3/13/2013 04:06:00 PM
:Posted by Alan Green, Software Engineer
:We have just announced on the Official Google Blog that we will soon retire Google Reader (the actual date is July 1, 2013). We know Reader has a devoted following who will be very sad to see it go. We’re sad too. 
:There are two simple reasons for this: usage of Google Reader has declined, and as a company we’re pouring all of our energy into fewer products. We think that kind of focus will make for a better user experience.
:To ensure a smooth transition, we’re providing a three-month sunset period so you have sufficient time to find an alternative feed-reading solution. If you want to retain your Reader data, including subscriptions, you can do so through Google Takeout.
:Thank you again for using Reader as your RSS platform.
Reader and Reader API were turned off soon after midnight, Pacific time, July 2.
== Post-shutdown message ==
:Thank you for stopping by.
:Google Reader has been discontinued. We want to thank all our loyal fans. We understand you may not agree with this decision, but we hope you'll come to love these alternatives as much as you loved Reader.
:Sincerely,
:The Google Reader team
:Frequently-asked questions
:# What will happen to my Google Reader data?'All Google Reader subscription data (eg. lists of people that you follow, items you have starred, notes you have created, etc.) will be systematically deleted from Google servers. You can download a copy of your Google Reader data via Google Takeout until 12PM PST July 15, 2013.
:# Will there be any way to retrieve my subscription data from Google in the future?No -- all subscription data will be permanently, and irrevocably deleted. Google will not be able to recover any Google Reader subscription data for any user after July 15, 2013.
:# Why was Google Reader discontinued?Please refer to our blog post for more information.
== Archiving ==
Archive Team's Ivan launched a heroic effort to retrieve historical feed data from the Google Reader API. Details can be found in the war room.
All WARCs have been uploaded to http://archive.org/details/archiveteam_greader. The total size is about 8800 GB (feed data + directory + stats).
We don't yet have a convenient tool to read a specific feed in the uploaded megawarcs. The master CDX index (see the CDX file format for interpreting each entry) gives metadata about which megawarc an archived file/feed/URL is in, and also the file's byte range, which allows seeking directly to it in the megawarc.
== Backing up your own data ==
* Main page - google.com/reader/
* Export some data via Google Takeout until July 15. Your ZIP file will contains subscriptions and starred items, but not tags. This can be imported into any of the Google Reader alternatives.
* Before it shut down, you could export *everything* with Reader is Dead
Category:Google
Gopher

The Gopher protocol is a "SmolNet" TCP/IP Application layer protocol designed for distributing, searching, and retrieving documents over the Internet. There's currently no standardized way to store archives of Gopher servers as WARC files. OpenNIC has a .gopher TLD intended for sites that only serve content over Gopher. Gopher+ is a backward-compatible extension that attempts to provide more information, so an archival effort should try to preserve all it can.
== Archives ==
* https://archive.org/details/quux-gopher-mirror
* https://archive.org/details/2007-gopher-mirror
* https://archive.org/details/gopher-may-2017.tar
* https://archive.org/details/rpodgopherhole
* Gopher archives torrents at iBiblio (broken link)
== Gopher sites == 
=== Online ===
* gopher://gopher.docfile.org
* gopher://gopher.floodgap.com
* gopher://port70.net
* gopher://gopher.4d2.org
* gopher://gopher.r-36.net
* gopher://sdf.lonestar.org
* gopher://gopher.quux.org
* gopher://distro.tube
* gopher://baud.baby
* gopher://eyeblea.ch
* gopher://gcpp.world
* gopher://gopher.black
* gopher://gopher.canich.net
* gopher://gopher.conman.org
* gopher://gopher.gluon.me
* gopher://gopher.libraryoferis.org
* gopher://gopherproject.org
* gopher://gopherspace.de
* gopher://gopher.xpenguin.club
* gopher://infinitelyremote.com
* gopher://invisibleup.com
* gopher://kagu-tsuchi.com
* gopher://medialab.freaknet.org
* gopher://raymii.org
* gopher://seedy.xyz
* gopher://shadowgate.org
* gopher://suika.erkin.party
* gopher://t3.at
* gopher://thinkmoult.com
* gopher://tildecow.com
* gopher://toromino.de
* gopher://zaledia.com
=== Offline ===
* gopher://gophernicus.org
* gopher://gopher.dna.affrc.go.jp:2347
* gopher://gopher.sm5sxl.net:3000
* gopher://moo.boo.com
* gopher://hal3000.cx
* gopher://gopher.undp.org
* gopher://kalb.ath.cx
* gopher://shamrockshire.yi.org
* gopher://atlas.chem.utah.edu
* gopher://hal3000.cx
* gopher://archive.ntp.org
* gopher://zzo38computer.cjb.net
* gopher://gopher.pcrpg.org
* gopher://pineapple.vg
* gopher://zcrayfish.dyndns.org
* gopher://gopher.lightlink.com
* gopher://gopher.sacrideo.us
* gopher://www.michaeleshun.4t.com
* gopher://george.hotelling.net
* gopher://gopher.viste-family.net
* gopher://86.43.88.90
* gopher://gopher.ludiclinux.com
* gopher://kiwidev.xyz
== Mailing Lists ==
* https://lists.debian.org/gopher-project/
== External links ==
* 
* https://github.com/jgoerzen/gopherbot
Governments/Germany

== Bundestag ==
* https://www.bundestag.de/
* http://webarchiv.bundestag.de/ | note = Own WinHTTrack-based regular snapshots of www.bundestag.de
* https://www.btg-bestellservice.de/
* Dokumentations- und Informationssystem für Parlamentsmaterialien (DIP)
* https://dserver.bundestag.de/ | note = Serves all Drucksachen (law drafts, motions, reports, etc.), also of the Bundesrat, but is not browsable; the DIP links to the files directly.
== Bundesrat ==
* https://www.bundesrat.de/
== Bundesregierung ==
* https://www.bundesregierung.de/
=== Bundeskanzlerin ===
* https://www.bundeskanzlerin.de/
== Bundesverfassungsgericht ==
* https://www.bundesverfassungsgericht.de/
== Federal law ==
* https://www.gesetze-im-internet.de/ | note = Official version of the federal law, currently valid version only
* https://www.buzer.de/ | note = Unofficial publication of the federal law with change tracking since 2006
* https://www.bgbl.de/ | note = Official publication of the Bundesgesetzblatt by the Bundesanzeiger Verlag, free to access but with many restrictions (e.g. no printing)
* https://offenegesetze.de/ | note = Unofficial publication of the Bundesgesetzblatt without restrictions
* https://www.bundesanzeiger.de/
* https://www.verwaltungsvorschriften-im-internet.de/
* https://www.rechtsprechung-im-internet.de/
Gutefrage

gutefrage (German "gute Frage" for "good question"), formerly gutefrage.net (until ), is a major German-language Q&A style site founded in . While there are no signs of it shutting down any time soon, questions may spontaneously be hidden from public view, meaning they require registration for viewing. This could impede future crawling should the site be in jeopardy.This question asked in July 2016 was still available a year later, but since at least 2018, it is behind a login wall.Another example: Asked in 2008, available until at least 2013, but behind login wall as of 2022.
It should be noted that "Yahoo! Answers" also ended up shutting down after working for well over a decade. 
== References ==
Category:Q&A
Category:German
Gyertyalang.hu

Gyertyalang.hu (literally candlelight.hu) is a special Hungarian website where mourning relatives and friends can light virtual candles for their deceased beloved ones.
The website seems to have been launched in 2008, and has retained its structure and style since then.
As of October 2020, more than 128,000 people have been commemorated on the website.
The service is operated by Port Kft., the company that also runs ingyenweb.hu.
HLTV.org

HLTV.org is a site dedicated to competitive Counter-Strike coverage with over a decade of material, some possibly dating back to 2002 or earlier. Though CS:GO material is more readily accessible, the CS 1.6 and CS:S material may be extremely tough to find elsewhere, if not irreplaceable.
== HLTV bought ==
HLTV.org announced on February 28th, 2020 that it was bought by sports betting company Better Collective for a sum of €34.5M.
Category:Video games
HTTrack options

Good options to use for httrack to mirror a large-ish site.
== Quick copy and paste ==
httrack --connection-per-second=50 --sockets=80 --keep-alive --display --verbose --advanced-progressinfo --disable-security-limits -n -i -s0 -m -F 'Mozilla/5.0 (X11;U; Linux i686; en-GB; rv:1.9.1) Gecko/20090624 Ubuntu/9.04 (jaunty) Firefox/3.5' -A100000000 -#L500000000 'YOURURL'
* ignores robots.txt
* allows for a queue of 500M unfetched URLS
* custom useragent
* pretty fast (uses several connections at once)
* will re-write links so they work offline
== A rundown of the previous options ==
* --connection-per-second=50: This allows for up to 50 connections per second.
* --sockets=80: Opens up to 80 sockets. If this gives you errors, lower this to 48.
* --disable-security-limits,-A100000000: By default, HTTrack attempts to play nicely with webservers, and tries not to overload them by limiting the download speed to 25kbps. On text-based sites this is normally good, but it becomes a hassle when the site is image-heavy. The first option disables the forced limit and the second one raises the limit to a large amount.
* -s0: Tells HTTrack to disobey robots.txt.
* -F: Sets the user agent.
* -#L500000000: Raises the maximum amount of links HTTrack fetches to 500M. Raise if needed.
* -n: This gets all nearby files (all files shown on a page), rather than only those on the domain name, which is HTTracks default behavior.
== Other options ==
* A full rundown of all possible options (including those on site structure) can be found here: https://www.httrack.com/html/fcguide.html .
NOTE: httrack runs java internally (I believe) and is limited to 2GB of ram. Not sure if a 64-bit version of it will allow for a larger crawl queue.
Category:Tools
Hacker News

Hacker News is a social news website for programmers and entrepreneurs. It is run by the startup incubator Y Combinator. 
== Vital Signs ==
Originally started as a demonstration of the Arc programming language by Y Combinator's co-founder Paul Graham, it has increasingly been getting more and more attention over the years. Probably won't disappear anytime soon without significant notice.
== Archives ==
* Dump of Hacker News stories and comments up to 2014-05-29 (compressed JSON files), made through one of HN's official API. Total: ~10.7GB uncompressed. More info from the .
* A more recent dump is described here: 
* Some comments and threads are also available on archive.is, including some that were lost during a .
== Site structure ==
* Official API docs available at 
** Older API at 
* "The site has a proactive attitude in moderating content, including automated flame and spam detectors." Comments made by hellbanned users are not viewable by other users, unless they have checked the "showdead" option in their profile.
** Example hellbanned user: 
** Dead comments can also be accessed through the official API. For example, 
* Threads and comments are accessible via the same URL: https://news.ycombinator.com/item?id=ID
** Each ID seems to be incremental. For example, is the very first thread. 
** The very first comment seems to be 
== External Links ==
* 
* 
* (includes basic Hacker News setup)
**
Hackercons

This is a project to collect audio/video from hacker conferences to archive.org.
=== Status ===
(This table is a work in progress)
Conference 	Done 	Missing 	Source
DEFCON 	15, 22 videos/audio (missing some metadata)	All but 15, 22	
Chaos Communications Congress	 	 	
CCC other events 	 	 	
Hack3rcon 	3, 5 	 	
Shmoocon 	2014 	 	
Notacon 	 	 	
BlackHat 	 	 	
Hackers on Planet Earth 	 	 	
Derbycon 	2, 3, 4 	 	
=== TODO ===
Uploaded talks that need to be moved to the hackercons collection:
* Irongeek's uploads
* DEFCON 22
* DEFCON 15
=== In progress ===
Works in progress:
* User:Mhazinsk is scripting DEFCON uploads
Hackerspaces

“A hackerspace (also referred to as a hacklab, hackspace, or makerspace) is a community-operated, often "not for profit", workspace where people with common interests, such as computers, machining, technology, science, digital art, or electronic art, can meet, socialize, and collaborate.https://www.lifehacker.com.au/2012/05/how-to-find-and-get-involved-with-a-hackerspace-in-your-community/”
hackerspaces.org keeps an extensive list of hackerspaces in their wiki, and also hosts a bunch of mailing lists for them.
== Europe ==
=== Germany ===
Some German hackerpaces are “Erfas” of the Chaos Computer Club and are listed there.
==== Aachen ====
===== CCCAC =====
* https://ccc.ac/
* https://wiki.aachen.ccc.de/
* https://git.ccc.ac/
* https://twitter.com/cccac
* https://chaos.social/@cccac
==== Regensburg ====
===== Binary Kitchen =====
* https://www.binary-kitchen.de/
* https://twitter.com/#!/binary_kitchen
* https://lists.binary-kitchen.de/mailman3/postorius/lists/
* https://chaos.social/@binary_kitchen
* https://botsin.space/@kitchenstatus
* https://github.com/Binary-Kitchen/
==== Saarbrücken ====
===== Hackerspace Saarbrücken =====
* https://www.hacksaar.de/ - 
* https://wiki.hacksaar.de/ - 
* https://lists.hacksaar.de/
* https://www.youtube.com/user/TechnikKulturSaarEv
* https://twitter.com/hacksaar
* https://www.facebook.com/TechnikKulturSaarEv
* https://social.saarland/@hacksaar
==== Unna ====
===== UN-Hack-Bar =====
* https://www.un-hack-bar.de/ (https://unhb.de/ points there, too)
* https://wiki.un-hack-bar.de/
* https://lists.unhb.de/postorius/lists/
* https://www.youtube.com/channel/UCAr5bd29PXNZl_H2VOnzZ-Q
* https://chaos.social/@un_hack_bar
* https://twitter.com/UN_Hack_Bar
* https://www.instagram.com/un_hack_bar
== Online ==
=== hackerspaces.org ===
“hackerspaces.org is an informal volunteer network of such [hacker]spaces, maintaining community services”https://hackerspaces.org/
* https://hackerspaces.org/ - small landing page
* https://wiki.hackerspaces.org/ - core of the project
* https://twitter.com/HackerspacesBot
* https://lists.hackerspaces.org/ - public mailing lists for hackerspaces
* https://signal.hackerspaces.org/ - inactive radio show “signal”; rss feeds are incomplete, but files are linked+hosted there