Significant roles in The Graph Protocol: The Indexer
Tags: The Graph, Dapps, Blockchain, Decentralization

In this article, the second in the series about roles in The Graph Protocol, we explore the Indexer role.

The Graph network is core infrastructure for Web 3.0, with a vision of fully decentralised applications that are robust and have unprecedented interoperability. It uses a model in which users make micro payments for decentralised services.

With nearly 200 indexers and 2000 curators across the globe, the Graph network is now being built to provide an open market place serving data for DeFi and Web 3.0 in a decentralised way. There are already over 2000 subgraphs, a number which is growing. It is paving the way for applications in the future to be serverless and enduring.

Blast off for mainnet!

Initially a hosted service processing more than 7 billion queries per month (a rate which is growing by 80% each month), it is now transitioning to a production ready decentralized network as it moves from testnet to mainnet. This is a process preceded by extensive design and development, the writing and auditing of thousands of lines of code, the establishment of a learning stakeholder community, multiple on-line education sessions and practical and competitive trials to prove understanding, adoption and capability.

One small step for man….

This is very much like a major but carefully considered leap. Indeed to quote Neil Armstrong as he put his left foot on the lunar surface in July 20, 1969: “That’s one small step for man, one giant leap for mankind.”

OK we may not actually be on a real-life space mission but to be actively involved in the dawn of the next generation internet is truly exciting! As this happens the community are actively addressing all the potential issues which could affect this next step.

The nature of indexing as part of the Graph process

One aspect of this transition is ensuring that the indexers are truly capable of performing their role.

As mentioned in our previous article, Indexers are just one of the important roles. Developers build and deploy Subgraphs describing how to use data from Web 3.0 data sources. Curators aim to deeply understand the Subgraphs and serve as navigators pointing out which have value and should be indexed. The Indexer participates in The Graph network by listening to the signals from the Curators, allocating their graph tokens (GRT) on appropriate Subgraphs and running nodes that serve queries on these Subgraphs in return for fees.

In this way Indexers provide a service to consumers who need to access the data held in Subgraphs. Part of the process by which a consumer requests queries on their chosen Subgraphs involves the software checking which indexers have the data they are interested in and then selecting one Indexer based on a number of parameters, including service performance and price. The query response is then sent to the consumer in return for a micro-payment.

An Indexer is able to serve data from a range of Subgraphs by selecting ones to allocate tokens to. The attractiveness and hence ultimately the allocation choice for each Subgraph is influenced by the curation signal provided by the Curators. Indexers are ideally aiming to respond solely to queries which provide adequate rewards.

Indexer Infrastructure capability

Indexers can only perform their role by operating a number of different types of nodes that index the data on Subgraphs and interact with the network to serve queries, organize payments and allocate stake. As data freshness and reliability strongly influence the choice of Indexer for each query, the quality and sophistication of this infrastructure is critical.

There are a variety of ways to deploy an Indexing operation. Basic, cloud-based, deployments using container technologies and 3rd party services to access historic blockchain data may initially suffice. At the other extreme are sophisticated and scalable solutions hosted in high-availability datacentes, such as the one LunaNova has designed and deploys.

The LunaNova solution for Indexing

The initial focus for The Graph will be providing Subgraphs for projects on the Ethereum network. Because of this LunaNova are self-hosting Ethereum archive nodes at each of our high-availability datacentres for performance and security reasons. This is no mean feat - each node takes well over 1 month to synchronize from scratch and consumes over 5 Terabytes of disk space! By self-hosting we eliminate reliance on 3rd parties who could throttle our query-rate or impose charges that may limit our services.

Subgraphs are stored in a PostgreSQL database, and our approach involves a sophisticated solution with replica and streaming-backup capabilities at geographically separate locations to boost reliability and uptime.

There are then various nodes that comprise The Graph software stack for indexing, querying and service provision. We run these in secure Virtual Machines (VMs) on our own hardware, enabling us to run multiple copies behind specially configured high-availability internal and external load balancers. This provides excellent horizontal scalability and redundancy.

Our architecture is deployed, configured and maintained by custom Ansible scripts for reliable operation. It is closely monitored by our purpose-designed high-availability Prometheus solution which uses multiple, redundant, routes to alert our 24/7 on-call sysadmins.

Indexer rewards

There are two types of rewards for Indexers: Query fees and

Indexer rewards.

Query fees from the rebate pool

Query fees are paid for by consumers on the network in exchange for receiving the answers to their queries. However instead of Indexers receiving fees directly for each query they serve, all fees are placed into a pool called the rebate pool. The share of this pool received by indexers is governed by some complex maths but essentially it aims to encourage indexers to allocate their tokens to a Subgraph in proportion to how many queries they are actually servicing on that Subgraph.

Indexer reward

Another incentive for Indexers is the indexer reward. The protocol generates this reward every epoch and it divides it up per Subgraph in proportion to the size of the curation signal each receives. This is then distributed to whichever Indexers are allocated on that particular Subgraph in proportion to their share of the total allocation.

Therefore, in addition to running reliable and performant systems, Indexers need to understand the intricacies of the query rebate function and how to best price their queries in order to make good returns for themselves and their delegators. As participants in The Graph’s Mission Control Incentivized Testnet since Phase 0, LunaNova have spent a considerable amount of time studying the educational material and experimenting to deeply understand it.

We are therefore well-placed to provide an Indexing service which is both reliable and offers competitive returns to our delegators.