API calls on Blockchain; best practices for data collection | by Patrick Collins | Towards Data Science

Most software engineers, when looking to get data from outside their programs, look to get data from API calls or HTTP GET/POST requests. Similarly, on a blockchain, people look to get data from an external API as well. This article will teach you how to do that! Let’s get going, here is step one:
Got you.
You actually can’t get data from an API call the same way you get data from normal software applications. However, in order for our smart contracts to do anything worthwhile, we would want them to interact with the outside world. So what gives?
The Ethereum blockchain was designed to be entirely deterministic. This means, that if I took the whole history of the network, then replayed it on my computer, I should always end up with the correct state.
Since the internet is non-deterministic and changes over time, then every time I replayed all of the transactions on the network, I would receive a different answer.
— Tjaden Hess on StackOverflow
Instead of the contract calling the API directly, we need to have it call an application that can interact with the outside world, and have it record its findings on-chain. These are called Oracles. Oracles are any application that interacts with data from the outside world and reports it back on-chain.
If we replayed the chain without oracles and used regular API calls, the API calls might have changed, and we’d get different results.
Thanks for the technicality definition update, but how do I get my data into my solidity application?
Ok ok, let’s get into what you actually came here for. In our example, we will be just trying to get the price of ETH in USD.
Now if you’re a TOTAL NOOB here with this Remix, solidity, Chainlink, and want a step by step guide, feel free to check out my more in-depth guide: Write your first solidity contract or Chainlink’s documentation. We all were imposters once :D
If you want to follow along and just press the buttons without having to choose your own oracles, you can follow along with all the code and deploy it with me in Remix here (code included in the hyperlink). HOWEVER, THAT CODE IS FOR DEMO ONLY AND YOU SHOULD NOT USE IT FOR PRODUCTION. You should put JobIDs and addresses as parameters. You can follow that along with us in Remix here for the more productized version. You’ll notice two of the steps are missing. This is intentional ;)
Go to the gists section. Be sure to send ROPSTEN LINK to your contracts after deploying them, otherwise, you’ll get a gas estimation error. You can get ropsten LINK here. DO NOT SEND YOUR ACTUAL LINK WHEN YOU ARE TESTING. Please. Please. Please. You can use the withdrawLink function if you accidentally send it LINK.
One other note: I tend to use oracle and node interchangeably, while technically not correct, the distinction between the two for this article is negligible.
The first approach most blockchain engineers start with is just finding an oracle technology, giving it the URL of our API call, and then having it do its thing of reporting the data on-chain for us to interact with. There are a number of oracles we can use to do this. We will be using Chainlink for reasons you’ll see soon.
To use Chainlink, we first have to pick a Chainlink oracle/node. These are independently operated smart contracts on the blockchain that allow you to interact with the world. To find one, we can go to node listing services like Linkpool’s Market.Link, and just choose a node. We then have to make sure that the node has a “http Get > uint256” job. Not every node can make URL calls and return a Uint256 (basically an Int), but most do!
The node we choose was a Linkpool node at the address in the ORACLE variable. All the magic happens in the requestEthereumPrice function. If you’ve followed Chainlink’s tutorials before, you know that this is the most simple way to get data using Chainlink.
This is great! We can get data. This is fine if you’re just testing and want to quickly develop your code, but please do not use this for a production smart contract. Why? It comes with one major issue:
You are pulling from one oracle and one data provider. This is not decentralized data! You need decentralized data in your solidity smart contracts otherwise you lose a major benefit of the application.
Now Linkpool is one of the most trusted Chainlink node services out there, but your application needs to be trustless. Not only that, but you’re also getting your data from one source as well, now you have TWO points of failure in your application. CryptoCompare and LinkPool are both massive liabilities that can be easily remedied. This is a rant worthy point that I will spare you from here, but let me drill it once more.
Your smart contract needs to never have a single point of failure, as that point can be bribed, hacked, be out-of-service at execution time, or many other reasons that leave your smart contract worthless.
So how do we fix this issue?
We’ve gone a step forward now and chosen 3 nodes that we will route our API calls to, greatly mitigating the issue from the naive approach. Now to get the price, we have added a median function, that takes the median of all the answers provided. This way if one or two give wildly different answers, we are more likely to still get the true value of ETH. You can see how added even more nodes would increase how little trust you have to put into each node.
Ideally, we’d also choose different data providers, but for simplicity, I’ve kept the same ones in the examples. Ideally continued, you’d like even more than 3 nodes, but at least now there are no single points of failure. At the moment, 7 or 21 nodes seems to be considered “decent”. Why 7 and 21? No idea.
So this is much better though! We now have decentralized data in solidity! Although, there is a lot of code that goes into this, is there a simpler way to route my API calls to multiple nodes?
I’m glad you asked.
If this part gets confusing for you, feel free to skip to 3.5
This one requires an extra step. You’ll notice in the remix link, there was one file that was just two lines of code.
pragma solidity ^0.5.0;import "github.com/smartcontractkit/chainlink/evm-contracts/src/v0.5/PreCoordinator.sol";
That’s it.
What this does, it that it creates what’s called a service agreement. This takes a list of oracle address and job IDs that you give it, and will fan out your API call to all those oracles, take the median, and everything for you, using the syntax of the naive approach!
Deploy theprecoordinator.sol, and in the GUI, add your oracle addresses, jobIDpayments (100000000000000000 = 0.1 LINK), and _minResponses. This is the minimum number of responses needed from each oracle.
Let’s say you have 5 oracles, and only 2 respond. If you set the minimum responses to 3, this would negate the rest of your smart contract. Which is ideal, since if we don’t have enough oracles respond, it will be less decentralized.
Once you launch a service agreement, you can use the address of the service agreement as your Oracle address, and the ID of the service agreement as the jobID. You’ll notice the syntax of this step is nearly identical to the naive approach. This is so that once you do all your testing with one oracle and are ready to move to production, you can just launch a service agreement and change the oracle and jobID, but keep all of the rest of your code the same.
It’s easy then to create new service agreements using different oracles if you decide some of the oracles you are using are giving you poor responses. You can do this without having to rewrite any of your existing code, just swap the IDs out for the new service agreement.
I don’t know about you, but I like it when I have to do less work to get more done.
This step is the most control you can have over a Chainlinked network. You can choose your nodes, data providers, and integrate and change them easily.
Note on data providers: Right now, the precoordinator can only send the same data package to every provider, meaning if the JSON output of different data APIs is different, you’d have to send different copyPaths. The way to send the same information while using different data providers is to wrap them all in an external adapter that formats their outputs to be the same. Stay tuned for updates on how to do this!
Sometimes you have to learn the hard way before you get the easier ways right?address PreCoordinator= 0x11db7845a757041F5Dad3fAf81d70ebAd394e9A2;
bytes32 constant patrick_service_agreement = 0xcb08d1dbcc1b0621b4e8d4aea6bafc79f456e12332c782b4258c7e83bcedb74c;
The PreCoordinator address above you can use on Ropsten to deploy arbitrary Service agreements. The patrick_service_agreement address you can use to send your httpget request to a few nodes I picked.
Ideally, you don’t store the PreCoordinator patrick_service_agreement , or payment in your code, but accept them as parameters of your functions.
I hope to get questions at the bottom if this confuses you.
A lot of oracles technologies have an out-the-box solution as well. Chainlink is no different. They have taken time to vet multiple nodes and data providers to give data feeds on a number of high use price feeds for anyone to use.
I’ve written about them before in Building Smart Contracts, Smartly, and they are the pinnacle of simplicity and decentralization.
You can find a list of the reference contracts they have on their feeds page. The ONLY part that makes it more centralized, is that Chainlink has chosen the list of nodes, this means that you have to trust a little in the node choosing ability of the Chainlink team. However, if you want a pre-boxed answer where maintenance of decentralization is outsourced to the Chainlink team, this is a huge advantage. Otherwise, you can always build your own Chainlink network of nodes to get your decentralized data in solidity!
The other best part of this is that it’s currently free, and is sponsored by awesome companies like Synthetix. I’m sure in the future this will change as it’s not a sustainable model.
Whew. There is a lot to cover here. I hope I have drilled the decentralization point. One of the main advantages of the blockchain is that we can have trustless applications.
Building a blockchain application with centralized data sources and oracles is like buying a bike so you don’t have to walk to work anymore, then proceeding to continue to walk to work with your bike strapped to your back.
You have invested the time in a solution and then proceed to still not solve the original problem, even though you have the solution right there.
Now that you have the tools and understand the importance here, let’s see you build something! For all new and experienced blockchain engineers, there are a ton of ways to join. AlphaVHack is a hackathon coming up for new engineers. You can join the Chainlink discord, build your first end-to-end blockchain solution with this awesome class from Udemy. Or just leave me a comment below asking how to find out more about anything above, talk soon!
Think I missed something? Were you able to follow the remix code? Don’t understand anything? Leave a question, comment, or insight below!
All opinions here are my own.
Follow me on twitter, medium, github, linkedin, discord, for more content and insights. Check out my latest article on Medium here!
#financerevolution #blockchain #ETH #chainlink #smartcontracts