A new perspective on Web Data Analysis
By: Revathi Suriyadeepan
Tags: Data Analysis, Graph, Network, Data Visualization, Web Analytics

NetworkX is a powerful tool for working with networks. But it didn’t suit the needs of this project, for the following reasons:

When the graph size is too big for example, the graph has 10k nodes, it takes a lot of time for rendering the graph

Rendering output is of low quality

Static hence we cannot interact with it

Why does the world need webG?

We do not have any tools for visualizing web pages as graphs because it is a relatively new idea. webG allows you to visualize the DOM tree of any web page as a graph. It currently supports 4 types of graph visualizations.

Force Directed

Random Tree

Radial Tree

Cluster

Given a URL of a web page, webG’s python backend scrapes the page, converts the DOM tree into a networkx graph, and passes it as a JSON string to the frontend for rendering, using El Grapho.

Features

A quick walkthrough of webG’s features.

Why Flask?

We have used Flask, a python web framework for quickly building web applications. Before looking into the details, let me explain why we’ve used flask. Why didn’t we build a simple javascript-based web page that runs on the client’s browser?

Browser security prevents a web page from making requests to a different domain. This restriction is called the same-origin policy. The same-origin policy prevents the site from reading sensitive data from another site. So, it is (almost) impossible to build a web page that scrapes other websites while running on the client’s browser. To overcome this restriction, we chose to build a web app using Flask. Cross-origin resource sharing (CORS) is a mechanism that allows restricted resources on a web page to be requested from another domain outside the domain. A Flask extension for handling Cross-Origin Resource Sharing (CORS) exists, making cross-origin AJAX possible.

Hello Flask!

Let’s first see a simple Hello World application using Flask.

In the example above, when the app is running and accessed at http://127.0.0.1:5000/ the user will get Hello World! as a response.

Dynamic web applications need static resources (CSS and JavaScript files) for programming their dynamic behaviour. Just create a folder called static in your package and it will be available at /static in the application.

To generate URLs for static files, use the special static endpoint name:

url_for(‘static’, filename=’style.css’)

The file has to be stored on the filesystem as static/style.css.

To render a template we have to use the render_template() method:

The code above renders the file hello.html placed under the templates/ folder.

El Glorioso Grapho

So, how does webG render the graph in the browser? Say hola to El Grapho, a powerful javascript library for visualizing large graphs. Why did we choose El Grapho? El Grapho is scalable — supports millions of interactive nodes and edges, high-performing and easy-to-use.

A note on my development environment

Initially, I used codepen.io for working on the project. It is simple and easy to use. It has a window pane for javascript, one for CSS, and one for HTML. It has an output pane that renders the HTML. It also has a console that helps with debugging. Codepen is great for quickly prototyping tiny project ideas. But for building a complete project we need to deal with more than one javascript, HTML, and CSS files. As the project idea naturally evolved into something bigger and bigger, I had to switch to VSCode, a full-fledged IDE for project development.

Program Flow

We’ve covered all the basics. Let’s take a look at how the code works.

Everything starts here. We have a text box for entering an URL. When we enter the URL and submit, the aptly named callback function button_callback() in graph.js is triggered.

The ajax call in the button_callback() function issues a GET request to localhost:5000/json. Flask responds to the GET request by calling the get_json() method. Notice the decorator on top of the function app.route(‘/json’, methods=[‘GET’]), which is a special flask construct that redirects the HTTP request to /json to the get_json() function.

Moving on to the internals of the python part of the code. We have a GET request from the client containing an URL as a parameter. We are supposed to respond with the DOM graph of the URL. webg.py handles this part. get_json() calls w2json() method of webg.py.

w2json() method traverses the DOM elements in the HTML soup of the given URL and creates a networkx graph, which is passed on to g2json() to convert it into JSON format. The JSON string is finally returned back.

The Flask server responds to the client with a DOM graph in JSON format.

Why JSON format? JSON (Javascript Object Notation) is the universally accepted text format for transferring data between a browser and a server. JSON is human-readable, easy to convert to javascript object and javascript objects are easy to convert into JSON without complicated parsing.

Now the client has the data that should be rendered as a graph on to the empty container on the browser page. The graph data is manipulated and transformed into a format compatible with El grapho using manipulateData().

The Model is stored globally. render_graph() renders it as an El Grapho graph.

toolTipFn() is a callback function that is triggered when the user places the cursor on a node. The function below renders the metadata of a selected node in a small pop-up window.

Applications

This tool would be useful to anyone working on Web Page Analysis, Web Scraping, Machine Learning, etc. Visualization of web pages as graphs offers intuitions about the organization of DOM elements. These intuitions could be built into a predictive model as features or they could lead to larger, powerful, and impactful ideas. It is essential to observe that by representing a webpage as a graph we are losing any kind of spatial information like the position of DOM elements in the browser. The position and size of DOM elements are crucial features that shouldn’t be neglected while building a predictive model.

Case Studies

1. Amazon vs Flipkart

Comparing Amazon and Flipkart’s cluster graphs, we arrive at the idea that Amazon advertises more (other) products on a product page compared to Flipkart.