NCC Digital Scholarship Workshop
Thursday, November 15, 2018

Introduction to web archiving using Archive-It
https://archive-it.org/
Instructor: Koko Fujita Howell
Japan Disasters Digital Archive http://jdarchive.org/en
Digital Content Lead/Web Archive Curator

Slide 1

What Archive-it can do
It is a powerful tool to save live websites, including YouTube,
which eventually disappear from the Internet at some point.
Websites usually last 1000 days to 5 years.
For example:

http://wayback.archive-it.org/2438/20130412201108/http://crms-setagaya.jimdo.com/
https://crms-setagaya.jimdo.com/

Slide 2

Archive-It Video Curriculum – Archive-It Help Center
Please watch these videos to understand how Archive-It works
https://support.archive-it.org/hc/en-us/articles/216489103-Archive-It-Video-Curriculum●

Getting Started○
Navigating Archive-It○
Administrative Functions○
Pre-crawl Scoping○
Test Crawls●

Post Crawl Analysis○
Getting the most from your post crawl reports○
Understanding your Hosts Report○
Quality Assurance
Advanced Training Webinars○
Advanced Scoping○
Archiving Video Content○
Archiving Social Media○Advanced Quality Assurance○
Access to Archive-It Collections●Slide 3

Data BudgetUnitAbbreviationStorage

Byte/Octet

B8 bits

Kilobyte

KB1024 bytes

Megabyte

MB1024KB

Gigabyte

GB1024MB

Terabyte

TB

1024BG

http://m3rabc.blogspot.com/ is about 1.8GB.
https://ameblo.jp/takahashi-photo/ is about 20~30GB

Monitor data budget usage in your account in order to prevent overspending.
https://partner.archive-it.org/1484/collections/11371/crawl/709014/hosts

Reference:https://support.archive-it.org/hc/en-us/articles/208000096-Monitor-your-data-budgethttps://physics.nist.gov/cuu/Units/binary.html
https://www.kingston.com/us/flash/storage_chart

Slide 4Tools for Archive-it
Web Browsers: Chrome and its bookmark bar
Safari is not suited because the address bar (location bar or URL bar) does not show full a URL.

YReference:https://doepud.co.uk/blog/anatomy-of-a-url
https://support.archive-it.org/hc/en-us/community/posts/360012216926-2018-PM-in-DC-Breakout-Group-3-Collaborative-web-archiving-

Slide 5

Tools for Archive-it (cont’d)Google Sheets to store your seeds (URLs)
It is suited for●●●●Easy export to your Archive-it collection seed page
Easy Quality Assurance (QA) through relatively simple programming
Easy search function for your task
Easy filtering system

Slide 6

A seed
A seed is any URL that tells the Internet Archive web crawler to capture information on the web.
Examples of seeds● an entire website (domain): http://www.soumu.go.jp/●

a specific part (directory) of a website:
http://www.soumu.go.jp/h30_hokkaido_iburitobu/●●

specific documents:
Html:http://www.soumu.go.jp/menu_kyotsuu/important/kinkyu01_000152.html

●
●
●
●

PDF: http://www.soumu.go.jp/main_content/000573592.pdf
Image: http://www.soumu.go.jp/main_content/000000014.gif
CSS: http://www.soumu.go.jp/main_content/aly.css
etc

References
https://support.archive-it.org/hc/en-us/articles/208331753-How-to-select-seeds-#What-exactly-is-a-seedhttps://en.wikipedia.org/wiki/Heritrix
https://en.wikipedia.org/wiki/Web_archiving

Slide 7

Scope・Scope is what the crawler will capture and what it won’t.
・Scoping refers to options for telling the crawler how much or how little of aseed to capture.
・Archive-It includes default scopes and seed- and collection-level scoping
options

Reference:https://support.archive-it.org/hc/en-us/articles/216489103-Archive-It-Video-Curriculumhttps://support.archive-it.org/hc/en-us/articles/208111686-Glossary-of-Archive-It-and-Web-Archiving-Terms
https://support.archive-it.org/hc/en-us/articles/208332843-Assign-and-edit-a-seed-type-

Slide 8

Scope (cont'd)
You have to determine which scope needed for needs.

●

The crawler can capture your seed URL and links within the seed site when you set the scope foryour crawl.
Archive-it default scoping rules●●

Standard: the crawler can capture URLs which have the same domain of the website
Standard +: the crawler can capture URLs which have the same domain of the website and one
page of external links
One page: the crawler can capture in one page of a website
One page +: the crawler can capture in one page of a website and the external links.

●●

Reference:https://support.archive-it.org/hc/en-us/articles/208332843-Assign-and-edit-a-seed-typehttps://support.archive-it.org/hc/en-us/articles/208001076-How-our-crawler-determines-scope

Slide 9Pre-crawl Scoping
Determining scope of crawl for your selected seedURL(s)an entire website: http://sada.la.coocan.jp/
Link: https://takeshiro.net/ is not in scope (domain is different from the example URL above)
Link: http://www.mctv.ne.jp/~takase/index.htm is not in scope (domain is from the example URL above)
Link: http://sada.la.coocan.jp/ta/ansei/tkhp02.htm is in scope (domain is the same as the example URLabove)
Embedded image: http://sada.la.coocan.jp/taketiti.JPG is in scope (domain is the same as the example URLabove)
References:
http://wayback.archive-it.org/11371-test/20181109103249/http://sada.la.coocan.jp/

https://support.archive-it.org/hc/en-us/articles/208001076-How-our-crawler-determines-scope
https://support.archive-it.org/hc/en-us/sections/201864583-Scoping-Crawls
https://support.archive-it.org/hc/en-us/articles/360015086931-Modify-your-collection-or-seed-scope

Slide 10Pre-crawl Scoping (cont’d)
Determining scope of crawl for your selected seedURL(s)Not in scope

http://www.mctv.ne.jp/~takase/index.htm
Slide 11

Adding Scope Rules to the collection and seed scope rule pages
Some seeds need to scope to capture images and block unwanted advertisements and videos.
Do Test Crawls!( do not cost data budget unless you save the data)

https://ameblo.jp/311hokenshi/
Needs special scope rule in order to capture images
https://ameblo.jp/311hokenshi/image
https://stat.ameba.jp/user_images/
Block images
S0.2mdn.net
Tpc.googlesyndication.com
Result of capture
http://wayback.archive-it.org/7472/20180524145206/
https://ameblo.jp/311hokenshi/
Reference:https://support.archive-it.org/hc/en-us/articles/208001106-Expand-t
he-scope-of-your-crawl

Slide 12Selecting seeds
https://partner.archive-it.org/1484/collections/7472/crawl/647275/seeds
発行物・提言書｜NGOを支援するNGO 国際協力NGOセンター（JANIC）
http://wayback.archive-it.org/7472/20180716055132/http://janic.org/activ/earthquake/issue/index.php

Seed: http://janic.org/activ/earthquake/issue/index.php
Scope → One page +Links within the web page
http://www.janic.org/MT/img/activity/JANIC_fukushima_kiroku.pdf
http://www.janic.org/MT/img/activity/ngo_recommendation.pdf
http://www.janic.org/MT/img/activity/positionpaper_nuclear.pdf
http://www.janic.org/MT/img/activity/statement_oda_japanearthquake.pdf
https://www.janic.org/MT/img/activity/JANIC_fukushima_kiroku.pdf
http://kodomira.com/news/HSK-release-all.html
http://www.shinhyoron.co.jp/978-4-7948-0913-1.html
Links are not in scope: (Links below have different directory fromhttp://janic.org/activ/earthquake/issue/)
You can also submit link URLs as separate seeds as One page/once(option private setting).

Slide 13

Seed scope rule (PDFs)
When you would like to capture pdfs within a web page you can also add scope
rule and do patch crawl after the initial crawl
2011年度：日調連会報：出版物のご紹介：日本土地家屋調査士会連合会の活動【日本土地家屋調査士会連合会】
http://www.chosashi.or.jp/activity/publications/kaiho/kaiho2011.html
Seed level scope
http://www.chosashi.or.jp/activity/publications/kaiho/img/kaihou
Or do patch crawl for this page
http://wayback.archive-it.org/7472/20181022125137/http://www.chosashi.or.jp/activity/publications/kaiho/kaiho2011.html

Slide 14

Seed scope rule
The websites listed below have specific scoping rules.

Facebook, Twitter, Instagram, Wix and Wixsite basewebsite, Blogspot, & YouTube videos
Please read the page👇
Reference:https://support.archive-it.org/hc/en-us/sections/201841373-Scoping-crawls-for-specific-types-of-sites

Slide 15

How to look at Archive-it account and collection pages
Examples:

https://partner.archive-it.org/1484
https://partner.archive-it.org/1484/collections/11371

NCCworkshop111518 has the account number 1484 following the domain
https://partner.archive-it.org/
You have to login to get your account URL
When you subscribe, you get your account page with specific number
https://partner.archive-it.org/xxxx
When you make collection you also get another specific number following /xxxx/collections/
https://partner.archive-it.org/xxxx/collections/++++
Slide 16

Archive-it collection pages
Frequently visited pages
Seeds: add seed URL
https://partner.archive-it.org/1484/collections/11371/seeds
Crawls: see non-test crawl and patch crawl results through Crawl IDs
https://partner.archive-it.org/1484/collections/11371/crawls
Collection Scope: add collection scope rules throughout the collection.
https://partner.archive-it.org/1484/collections/11371/scope
Wayback QA: find missing URLs for crawl patch. Usually URLs listed in the page are found
Under Queued or Out of Scope column in a host page within the Crawl ID pagehttps://partner.archive-it.org/1484/collections/11371/qa

Slide 17Test crawl results・Test crawl results are available only through “Crawl Reports”.
https://partner.archive-it.org/1484/collections/11371/crawls

・Go to crawl page and click “Test crawls” among horizontal
navigation menus,
https://partner.archive-it.org/1484/collections/11371/crawls/test

・Click test crawl id number, then the crawl overview page opens.
・Then, click “seed” to see your seed lists.
・Then, click under Wayback links.
You have to click many times to get test crawl results, so I recommend thatyou make a crawl report spreadsheet on your own.

Slide 18

Crawl report table using Google Sheets (option)
Nov 2, 2018

Nov 1, 2018696897696799Test

https://partner.archiv https://partner.archiv https://partner.archiv
e-it.org/1131/collecti e-it.org/1131/collecti e-it.org/1131/collecti
ons/7472/crawl/6968 ons/7472/crawl/6968 ons/7472/crawl/6968
97/
97/seeds
97/typesno data yet

One-Time

https://partner.archiv https://partner.archiv https://partner.archiv
e-it.org/1131/collecti e-it.org/1131/collecti e-it.org/1131/collecti
ons/7472/crawl/6967 ons/7472/crawl/6967 ons/7472/crawl/6967
99/99/seeds
99/typesverified

Note: you have to program the google sheet to open the desired page

Slide 19

Unfortunate results
Incomplete capture examples
気仙沼信用金庫http://wayback.archive-it.org/7472/20180904003005/http://www.shinkin.co.jp/kshinkin/

Earthquake triggers landslides in Hokkaido, Japan – in pictures |World news | The Guardian
http://wayback.archive-it.org/7472/20180928234027/https://www.theguardian.com/world/gallery/2018/sep/06/earthquake-landslides-hokkaido-japan-in-pictures

Slide 20

Unfortunate results & the Wayback calendar page
(Blocked by robots)

Wayback> is connected to the URL’s calendar page, which shows the URL’s
captured dates.
http://wayback.archive-it.org/7472/*/http://reroots.blog.shinobi.jp/Category/1/

Unblock the domain in the collection or seed scopepages
Reference:https://support.archive-it.org/hc/en-us/articles/208001096-Avoid-robots-txt-exclusions

Slide21

The Main Point of This Presentation
Prior to Final Crawl
●
●
●
●

●

Check your seed formatting (how a web page is constructed).
Consider your seed types and decide the scopes: Standard, Standard +, One Page, One Page +Do Test crawls(Test crawls do not use up the data budget unless you save the data)
Check the test crawls and add scoping rules (collection- and seed-level scoping) if necessary.
(Before you try to do a second test crawl for the same seed, I recommend that you delete the testcrawl previously done.)
Do the actual crawl for the seed.

Slide 22

Introduction to web archiving using Archive-It
Instructor: Koko Fujita Howell
Japan Disasters Digital Archive http://jdarchive.org/en
Digital Content Lead/Web Archive Curator

Thank youTime for the Archive-it experience!

Slide 23