Intro to Open Database for Geoscience Computing: Part 1 of 2 | by Yohanes Nuwara | Towards Data Science

Nowadays, open geoscience data are very important for researches, benchmarking, and project purposes. They are so useful that people working in the energy industries (oil and gas, and geothermal) and in academia utilize these datasets to do benchmarking on their methodology, and also as reproducible teaching material for higher university students. At the same time, not all institutions have access to a wide range of industrial commercial software or applications for the computation and processing of these datasets. The purpose of this article is to introduce a tutorial on how to make the most of the available open databases.
I have compiled at least three massive open geoscience databases on the internet, which I believe everyone now could access them “under the same roof”! Here are the three open geoscience databases.
Public geoscience Data in Google Drive
SEG Wiki Open Data
Geothermal Data Repository (GDR) OpenEI
Public geoscience Data is an open database available as a Google Drive, created by Peter Amstrand. This database contains a wide range of reproducible datasets, such as the seismic 3D data of Netherlands F3 and Canning projects, well-log files of North Sea projects compiled by GEOLINK, variety of geoscience images used for training and testing datasets in machine learning researches, and many more.
SEG Wiki Open Data is a catalog of available open geophysical data owned by the Society of Exploration Geophysicists (SEG). This catalog contains more than 30 geophysical datasets, such as offshore and onshore seismic data and well-log files, the New Zealand 3D projects, geophysical synthetic benchmark models such as BP Benchmark Models, Marmousi Model, and KFUPM-KAUST Red Sea model, gravity-magnetic data, and topographic-bathymetric data.
Geothermal Data Repository or GDR OpenEI is an open geothermal energy data portal provided by the United States Department of Energy (DoE) and developed by the National Renewable Energy Laboratory (NREL). As it is apparent for the name, this data portal mainly focuses on geothermal and hydrothermal exploration data, for instance, the FORGE Project data near Milford in Utah.
The use of Python as an open-source programming language enables people to develop a variety of powerful programs, as its utilization in geoscientific purposes is also inevitably very well recognized. Python programming language is also now becoming a “communication language” among developers. With no doubt, people with limited access to commercial software could develop their own programs using Python.
In addition, running in a cloud service is a seamless opportunity to increase efficiency in computing. One does not require to take up large space for memory in the local computer. Google Colaboratory (or Colab) is a research initiative of Google to make computing in Python possible in the cloud. The type of Python shell we are working in Google Colab is an IPython notebook.
This article consists of 3 tutorials, started from how to access the Public geoscience Data Google Drive, then how to unzip and open simple files, and finally how to access the SEG Wiki and GDR OpenEI directly from the website to Google Colab.
Open the Google Drive Public geoscience Data link. Once you clicked the link, you will have a copy of that database stored in your Shared with Me directory. Your next move is to transfer Public geoscience Data from your Shared with Me directory to your My Drive. You may not know how to perform this action in Google Drive. The following GIF animation shows you the way!
The Public geoscience Data is now stored in your My Drive directory. Make sure it is stored by visiting your My Drive directory and search for the Public geoscience Data folder. Now open the folder. You will find 12 folders and 2 files inside.
However, not all folders and files inside that Public geoscience Data contains geoscience datasets for your further processing, analysis, and computation. For instance, the report for images folder contains geoscience documents (in PDF and image format) published by the University of Stavanger and the University of Oslo. These documents should be very valuable for research references, but not for computations (seismic processing, well log analysis, et cetera). If you are curious about the content, here is the list that I made.
In the above list, you find some folders contain ZIP folders and some simple files such as images in format PNG and JPG. In Tutorial 2 of this article, you will learn how to unzip files and open these images in Google Colab. However, the tutorial on how to access big files (which are the geoscience datasets) such as the seismic data SEGY or SGY and well-log data LAS files, won’t be covered here since we require a very long discussion. But, don’t worry, we will cover this up in the next Part 2 of this tutorial series. Later in Part 2, we will cover more exposure on Python codes.
Another tutorial article on the way: Intro to Open Database for Geoscience Computing, Part 2 of 2
After sorting out, hence, I have screened 6 folders that contain datasets, as follows:
Canning 3D TDQ seismic
Dutch F3 seismic data
GEOLINK North Sea wells
Poseidon seismic
Core images
48 well composite logs
Next, if you checked the detail of just one of the 6 folders, for instance in Canning 3D TDQ, you will find a seismic data file named canning3d_GDA94_UTM50s.sgy that has a file size of 103 gigabytes! You aren’t surprised if you are in the oil and gas industry because in fact, normally does it so as most seismic data, the file size is way very large. It won’t be a wise decision to download this very large seismic data on your local PC. That’s why I introduce you to an effective way to open this file in Cloud. The Cloud service that I mentioned here is Google Colab.
So now let’s go to Google Colab. Visit this link to redirect you to Google Colab. At the very first window that pops up when you visit Google Colab, click New Notebook to create a new IPython notebook.
Then you will be redirected to a your new notebook. Now, you already have a new blank notebook, where you will put codes to open the Public geoscience Data. The following is the structure of a notebook.
On top of your new notebook, you see Untitled.ipynb. It is the name of your notebook, so change it to your preferred name, but remember not to remove the extension .ipynb, because it’s the typical extension of IPython notebook. For instance, name our new notebook as Public-geoscience-Data.ipynb.
You see also Code. The Code is used to create a new cell code in the notebook, so whenever you create a code script, click on it to add a new cell. For warming up, you could create some codes, as it is shown below. To run your code, click the Play button on the left of the cell, or simply do CTRL + ENTER.
At the far left of your notebook, you find also three symbols. The bottom-most symbol is Folder, where you could navigate the directories which you are working on now. By default, if you clicked it, there is only one folder named sample_data. It is created for you by Colab but here, we don’t use it.
Now, let’s start to access your Public geoscience Data folder in Google Drive to your Google Colab. Add a new cell by clicking the Code button, and run the following script.
from google.colab import drivedrive.mount('/content/drive')
The following message will appear. The message provides a Google Drive link. Click that link and follow the instructions you need to access your Google Drive account. A new link will again be provided to you, copy it, and paste the link back to your notebook cell inside the “Enter your authorization code:” Then, press ENTER. Please wait until the authorization process is finished.
If you have finished the authorization, you could check your Google Drive folders available in the Folder button (previously discussed). Click REFRESHif you could not find it. Then navigate through your Google Drive folders to find the Public geoscience Data. This is the path of that folder.
"/content/My Drive/Public geoscience Data"
The folder should be there. If you did not find that folder, you might not succeed to move the Public geoscience Data from your Shared with Me to your My Drive. In that case, you should check out the above tutorial on how to do so again.
Another way to ensure that Public geoscience Data is in the path, run the following script.
cd "/content/drive/My Drive/Public geoscience Data"
If it returns fine, then fine, no problem. If a message comes out “No such file or directory”, it means you need to make sure you have successfully moved the folder to your My Drive.
Next move, see the content of the folder. You have 2 ways to do so. The easiest way is to navigate to the Folder section at the far left of your notebook and expand the folder, just like when you operate in your PC. Another elegant way is by running the following script.
ls "/content/drive/My Drive/Public geoscience Data"
You will find the following contents.
The script ls is used to inspect a folder, whereas the previously mentioned cd is used to go to a certain folder in our working directory. It’s called a Bash script, a Command-Line programming language in Linux. This one is an interesting topic, we don’t need to focus on that one at this time being!
Congratulations! You now have your open database in your Google Colab! As it is previously discussed, only 6 files inside this Public geoscience Data will be considered datasets.
Further, you will see if you look into some folders (Recall: Table 1), such as the GEOLINK North Sea Wells. Again run ls to inspect the files inside this folder.
ls "/content/drive/My Drive/Public geoscience Data/GEOLINK North sea wells with Lithology interpretation"
There is one ZIP file named GEOLINK_Lithology and wells NORTH SEA.zip, also, an image file in PNG format named Lithology code in the well.png. In the next Tutorial 2, you will learn how to unzip this file and open an image in your Google Colab notebook.
The advantage of using Google Colab is viable, you could unzip a file without downloading it into your local PC and without using any unzipper program. To the above GEOLINK North Sea ZIP file, run the following script to unzip this file.
!unzip '/content/drive/My Drive/Public geoscience Data/GEOLINK North sea wells with Lithology interpretation/GEOLINK_Lithology and wells NORTH SEA.zip' -d '/content/GEOLINK North Sea'
It will take some time to unzip the files. When it is finished, the unzipped file will appear in a new folder under your working directory /content, called GEOLINK North Sea.
The above script could be broken down into four syntaxes: !unzip ‘zip_file_path’ -d ‘to_new_folder’. The first syntax !unzip instructs Colab to unzip the file, the second syntax ‘zip_file_path’ is the path of the zip file, the third syntax -d tells Colab to store the unzipped file into a new directory, where the new directory path name is in the fourth syntax ‘to_new_folder’.
Congratulations! You now know how to unzip a file. Because the unzipped files are now stored in a new directory, run ls to inspect the contents.
ls '/content/GEOLINK North Sea'
You will find approximately 200 well-log LAS files individually inside.
These are the wells drilled in the North Sea. Again, the tutorial on how to open these LAS files will be covered in the sequel article in Part 2.
We will now focus on another task, opening a simple file such as a PNG or JPG image. If you go back to the original GEOLINK North Sea Wells folder, you will find an image file named location of geolink wells.png. It is the location map of these wells.
We will use a Python library called Pillow or PIL. First off, we need to import a module from PIL named Image in our Colab notebook (PIL has been already in Colab, so we do not need to install.)
from PIL import Image
Next, run the following script to open the image. The image file path is /content/GEOLINK North Sea/location of wells.png. We use the open function from Image module to do so.
img = Image.open("/content/GEOLINK North Sea/location of wells.png")img
When you run the script, an image will appear in your Colab notebook! As you have already guessed, this is the North Sea map that contains the well locations. This is an offshore field near Snorre Field.
Congratulations! You already know how to open an image in Colab. Until this discussion, you know enough how to access some datasets from Public geoscience Data Google Drive. Of course, there are still many more untouched files discussed in this Part 1 article, such as seismic data and well-log files, so stay tuned for the Part 2 article. At least, you know how to access a geoscience database in Google Drive directly to Google Colab.
Next, in Tutorial 3, we will discuss on how to access the other open databases, namely the SEG Wiki Open Data and GDR OpenEI.
To access both these databases, we do not need the above Google Drive workflows anymore! We could stream and download the dataset directly from the webpage to our Colab notebook.
We need to peek what both of these open databases webpage look like. First off, we will visit the SEG Wiki Open Data webpage.
In the contents, you will see 30+ open datasets. We recommend visiting the content link first of the data that you would like to access Google Colab. You will find the details of the data and click the associated link. For instance, visit this link Stratton 3D seismic dataset.
Scroll down in the Stratton 3D webpage to section How to download, there are files with links. For now, we will use in the third row, the 3D filtered migration file, right-click on it, and “Copy link address”.
The link has been copied, now go back to your Colab notebook, paste the copied URL in the following script, and run the script. Remember to paste the copied URL inside the single quotation mark.
!wget 'your_copied_URL' -P '/content/Stratton/seismic'
Here, we use !wget syntax to download the dataset in the URL. The next syntax -P is passed to store the downloaded data into a new folder, which named in the next syntax, '/content/Stratton/seismic’. This !wget syntax is universal, meaning that you could download any dataset in a form of URL.
As we noticed while downloading the dataset URL, the file size is 423 megabytes! Imagine we download this dataset into our local computer, it surely will take up large space.
If we navigate to the newly created folder seismic inside Stratton, we will see a seismic data named Stratton3D_32bit.sgy. In Part 2 article, we will discuss how to open this seismic data.
Congratulations! You now know how to download a dataset from the SEG Wiki Open Data directly to Google Colab. The same way of using !wget we will also apply to another GDR OpenEI database.
Next, let’s visit the Geothermal Data Repository (GDR) OpenEI webpage. In the first encounter, you need to browse a dataset with a certain keyword to search. Just type gravity forge. We will use an instance of 3D gravity data of FORGE site near Milford in Utah.
Visit the Utah FORGE: 3D Gravity Data that appears in the search webpage. Unless you find it, just browse the complete name in the “Search” block. Inside the page, scroll down to find the ZIP file FORGE_3D_gravity.zip, right-click on the “Download”, and similarly “Copy link address”.
Again, go back to the Colab notebook and this time we will leave it to you! If you still remember (we are sure you do), use !wget to download the URL and then specify the target folder.
!wget 'https://gdr.openei.org/files/1144/FORGE_3D_gravity%20(1).zip' -P /content/FORGE
You now get the downloaded file in the newly created folder FORGE. And as you realize, it is a ZIP file. You already know how to unzip it for sure. Run the following script.
!unzip '/content/FORGE_3D_gravity (1).zip' -d '/content/FORGE'
Congratulations! You have accessed some datasets from GDR OpenEI using !wget
Throughout the first part of this introductory article series on Open Geoscience Computing using Python in the Google Colab cloud environment, we have discussed three tutorials that cover basic workflow to access the open datasets.
For wrap up, I created a Colab notebook that contains all the codes discussed in this tutorial. Open the notebook here.
Along with this article, I also made a GitHub repository that contains step-by-step tutorials about Open Geoscience Computing. Visit this link to start the journey.
Last but not least, it is highly recommended for anyone who is interested to use open datasets to cite these data with respect to the data owners. The Citation Wiki is available on my GitHub wiki page. The citation content will also be frequently updated.
Enjoy your Open Geoscience Computing, and stay tuned for Part 2!
Reach my GitHub: github.com/yohanesnuwara
Reach my email: ign.nuwara97@gmail.com
Reach my LinkedIn: https://www.linkedin.com/in/yohanes-nuwara-5492b4118/