# From Navicrawler to HyBro: a brief history of webcrawlers for social sciences

Benjamin Ooghe-Tabanou

<figure><img src="image_1.png" /></figure>

To cite this version:

Benjamin Ooghe-Tabanou. From Navicrawler to HyBro: a brief history of webcrawlers for social sciences. FOSDEM 2021 - Open Research Tools and Technologies devroom, Feb 2021, Brussels, Belgium. hal-03904098

HAL Id: hal-03904098

https://sciencespo.hal.science/hal-03904098

Submitted on 16 Dec 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution - ShareAlike 4.0 International License
<figure><img src="image_3.png" /></figure>

# From Navicrawler to HyBro: a brief history of web爬lers for social sciences

## FOSDEM'21

supposedly in Brussels, actually on the web!

February 6th 2021

Benjamin Ooghe-Tabanou (@boogheta)

Sciences Po médialab (@medialab_ScPo)

SciencesPo
MÉDIALAB

<figure><img src="image_4.png" /></figure>

DIME - SHS
FORCCAST

<figure><img src="image_5.png" /></figure>
# Webcrawlers? What is that?

> « Web crawling is the process of building a collection of webpages by starting with an initial set of URLs and recursively traversing the corresponding pages to find additional links. »

Gabe Ignatow & Rada Mihalcea, in: Text Mining: A Guidebook for the Social Sciences

But why would anyone want to do that?

- create an index for a search engine (like... you know...)

- extract contents or structured data (like scraping)

- perform some text mining

- study network communities through hyperlinks
# Crawling the web for social sciences?

> « **Hyperlink** » is at the core of the Web's architecture
> → charged with meaning and structure

> « The texts are **linked together** in a way
> that one can go from one concept to
> another to find the information one wants.

> The network of links is called a **web**. [...]

> The texts are known as **nodes**.

> The process of proceeding from node to
> node is called **navigation**. »

Tim Berners-Lee, 1990, WorldWideWeb: Proposal for a HyperText Project

<figure><img src="image_6.png" /></figure>

> « A hyperlink is a **manifestation of intention**. By linking one page to another, one piece of text to another, **people intend** to do particular things. »

Ryfe, Mensing, & Kelley, 2016, What is the meaning of a news link?
# Some examples of webcrawling research outputs

**Corpus web sur le changement climatique : le changement climatique est-il anthropique ?**

Le réseaux représentent les actions faites qu'elles sont reçues par les web perçus par des humains. Les réseaux les plus cités sur le corpus ont une plus grande ballade. Le couvert des liens est composé de facteurs qu'en est d' nature.

**Web corpus on climate change: anthropogenic climate change**

The network represents how actors interact on the web. The most active actors have the largest representation. Links are colored according to their source.

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

Most hyperlinks stem from the unreliable and aim at the reliable resources

Each bloc's surface is proportional to the count of websites. The color code si the same as the "Décodex".

<figure><img src="image_9.png" /></figure>

The thickness si proportional to the normalized link density*

<figure><img src="image_10.png" /></figure>

## Double Dating Data: Climate change on the web

<figure><img src="image_11.png" /></figure>

## Visual Network Exploration for Data Journalists

<figure><img src="image_12.png" /></figure>

## Privacy Web Corpus Datascape

## Unfolding the multi-layered structure of the French Mediascape
# Different possible crawling strategies

* Focus crawling

* In depth snowballing

* Selective crawling with iterative curation

* etc.

<figure><img src="image_13.png" /></figure>

<figure><img src="image_14.png" /></figure>
Windows desktop application

Latest version in 2016

http://socscibot.wlv.ac.uk

<figure><img src="image_15.png" /></figure>

<figure><img src="image_16.png" /></figure>
# Navicrawler (by WebAtlas & eDiasporas, France)

## Firefox (≤ v3.5.1) addon

https://github.com/medialab/navicrawler (archived)

<figure><img src="image_17.png" /></figure>

Web Atlas

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>
# IssueCrawler (by Digital Methods Initiative, Netherlands)

## Web application

## Different crawling strategies offered

https://www.issuecrawler.net

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>
# VOSON (by Research School of Social Sciences, Australia)

## Web application

http://vosonlab.net/VOSON

<figure><img src="image_22.png" /></figure>
# Hyphe (by médialab Sciences Po, France)

Web application

Dynamic web entities + Iterative focused crawling

https://hyphe.medialab.sciences-po.fr/demo/

https://github.com/medialab/hyphe

<figure><img src="image_23.png" /></figure>
# HyBro (by médialab Sciences Po, France)

Hyphe Browser → crawl with Hyphe, but while browsing like NaviCrawler
Linux / MacOS / Windows desktop application (based on electronJs)
v2.0 just released with complete redesign! https://github.com/medialab/hyphe-browser/releases/

<figure><img src="image_24.png" /></figure>