Web Scraping CryptoCurrency price and storing it in MongoDB using Python

29 Dec, 2020
Let us see how to fetch history price in USD or BTC, traded volume and market cap for a given date range using Santiment API and storing the data into MongoDB collection.
Python is a mature language and getting much used in the Cryptocurrency domain. MongoDB is a NoSQL database getting paired with Python in many projects which helps to hold details that got retrieved from Python Programs. PyMongo is a Python distribution containing tools for working with MongoDB and it is a very convenient way  to work with MongoDB from Python to do Create/Update/Delete/Read operations easily.
Let us see the code in using Santiment’s API in getting cryptocurrency prices for a given date range
Few Examples of cryptocurrency are :
bitcoin
ethereum
ripple
bitcoin-cash
litecoin
eos
cardano
stellar
neo
iota
In the place of id, we can pass bitcoin/ethereum or any cryptocurrency name which the below code can parse and get the dataFor valid cryptocurrency name, data will be retrieved properlyIn the place of from_date and to_date valid dates in valid date formats in yyyy-mm-dd pattern need to be given. For understanding purpose, it is given to take 7 days of data. We can able to get 1 month old data too. API call can able to get data for the given cryptocurrencies in the given date range or else if not available (due to invalid cryptocurrency name/invalid date range specification), we can catch the errorsIn the below program, for easier understanding, taking bitcoin and ethereum in “IndexCoins.idx” file and hence in the place of id, they are passed
In the place of id, we can pass bitcoin/ethereum or any cryptocurrency name which the below code can parse and get the data
For valid cryptocurrency name, data will be retrieved properly
In the place of from_date and to_date valid dates in valid date formats in yyyy-mm-dd pattern need to be given. For understanding purpose, it is given to take 7 days of data. We can able to get 1 month old data too. API call can able to get data for the given cryptocurrencies in the given date range or else if not available (due to invalid cryptocurrency name/invalid date range specification), we can catch the errors
In the below program, for easier understanding, taking bitcoin and ethereum in “IndexCoins.idx” file and hence in the place of id, they are passed
Example :
For the following input
For bitcoin above API call, returns data as below
Similarly for ethereum also, we will get value
Retrieved value has to be stored in MongoDB
Database : geeksforgeeks
Collection : Coll_santiment_Price
As we are going to take for 7 days of data, it has to be given in a loop
In MongoDB, _id is the default column which can be got by ObjectId(). Our process has multiple rows where each and every row is identified by means of “cryptocurrencyname” and “time”. In our code, we are having it as “id” and “time” respectively.
Let “Coll_santiment_Price” be created with 3 columns namely _id, id and time first. Then upon successful execution of API call, for the id and time, let API call output namely (verify column name from bitcoinprice.png) ‘priceBtc’, ‘priceUsd’,’volume’ and ‘marketcap’ are updated in a loop.
Code Implementation in Python
Python3
# importing the modulesfrom datetime import date, timedelta, datetimefrom pymongo import MongoClientimport sanfrom bson.objectid import ObjectId  client = MongoClient('mongodb://localhost:27017/')db1 = client.geeksforgeeksdata = {}  with open("IndexCoins.idx") as openfileobject:    for line in openfileobject:                  # One by one File's cryptocurrency names are read        id = line.strip()         try:            print(id)                          # Collecting the data for 7 days and hence range(7) is taken            for idx in range(7):                daystr = str(date.today() - timedelta(days = idx))                  # Coll_santiment_Price collection documents need to be created                # It will have columns namely "_id", "time", "id",                # "priceBtc", "priceUsd", "volume", "marketcap"                data['id'] = id                data['time'] = daystr                  # _id column for unique key and can be generated by using ObjectId()                data['_id'] = ObjectId()                                  # Initially create 3 columns in collection as rest of the                # columns are updated after running santiment api call                              db1.Coll_santiment_Price.insert(data)             try:                # Santiment API call to get Cryptocurrency prices                daa = san.get("prices/" + id,                              from_date = "2020-08-20",                              to_date = "2020-08-27",                              interval = "1d")                                  # API call output and for bitcoin it is given in                # https://media.geeksforgeeks.org/wp-content/uploads/20200827191739/bitcoinprice.png                print(daa)                              except:                print("URL error")                continue;                              # 7 days output            for idx in range(7):                                  daystr = str(date.today() - timedelta(days=idx))                  # API call output for the above chosen date                row = daa.loc[daystr]                                  # priceBtc, priceUsd, volume and marketcap are                # collected and stored in seperate variables                priceBtc = row['priceBtc']                priceUsd = row['priceUsd']                volume = row['volume']                marketcap = row['marketcap']                print(id, daystr, priceBtc, priceUsd, volume, marketcap)                try:                    # Update the collection with above details against                    # cryptocurrency id and time  i.e. bitcoin and 2020-08-27                    db1.Coll_santiment_Price.update(                        {'time': daystr, 'id': id},                        {"$set": {"priceBtc": priceBtc,                                  "priceUsd": priceUsd,                                  "volume": volume,                                  "marketcap": marketcap,                                  }                         },                        upsert = True                        )                except Exception as e:                    print(e)        except:            print("Error")
Sample Output : (For bitcoin)
Sample Output : (For Ethereum)
Santiment API calls for getting prices are very much used across cryptocurrency projects. As we are getting historical data, for data analytics, this is very much useful. Moreover, it is freely accessible and hence any beginner-level projects can use it without any issues. Price variations are not much varying and hence well set for demo and small scale projects.
Python web-scraping-exercises
Python-mongoDB
python-utility
Web-scraping
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.