Retrieve and Process hyperspectral field measurements | Towards Data Science

As technology evolves rapidly, remote sensing is finding its many uses across different subjects, from forest fire mapping to water quality assessment, 3D surface modeling and many others. The advantages of obtaining physical characteristics of an area by measuring its reflected and emitted radiation at a distance are many. However, field measurements are as important as the remote sensed data, in order to conduct researches that will make it possible to correlate what is “seen” at a distance with the ground truth reality. In the case of multispectral remote sensed images, it is important to conduct field measurements using hyperspectral sensors that will permit to calibrate the final results.
It was the beginning of 2021 when my thesis supervisor told me that we needed to recover some hyper spectral field measurements. These measurements have been done during various ancient terrain campaigns across the world. The objective seemed quite simple. I should retrieve the measurements from 3 field sensors (irradiance and radiances — upwelling and sky’s) to calculate the final reflectance (eq. 1). All field measurements were done by using well known sensors, and the results were stored in Microsoft Access files (.mdb). As there was a software specific to manage the .mdb files from the manufacturer of the sensors, called MDSA XE, I thought it would be a “piece of cake”. And I couldn’t be more wrong.
Germans are very well known for their industry and the quality of their equipment. That’s without a doubt. However, when it comes to systems, user interface or usability, it seems that there’s always something missing. And, in the case of the MSDA, a lot was missing. Even a simple task such open a file is a weird operation that involves restarting the application. I couldn’t rely on that for my task.
For that, I’ve developed an “unofficial” framework called RadiometryTrios, that is available for Python and that will make things a lot easier, not only to access this kind of data, but also to create an entire database from them.
As explained, this framework is responsible for manipulating hyperspectral radiometry measurements from field campaigns, using the TRIOS equipment.
The manipulation involves:
Extracting radiometry measurements from .MDB files in the MSDA/TRIOS format.
Analysis of the radiometry, data cleaning, and graph plotting
Wavelengths Interpolation
Reflectance generation (from radiance and irradiance measurements)
Input/Output from/to different formats
The schema of the RadiometryTrios framework is displayed in Figure 1.
The classes that are declared in the package are:
BaseRadiometry: this class holds some common functionalities that are used across the other classes such as plotting and interpolating functions. The methods for this class are declared as “static” and can be used as simple functions.
TriosMDB: this class is responsible for connecting to the access .mdb file and extracting the radiometries from it. It is possible to filter values so that the extraction corresponds to the interested measurement.
Radiometry: this class represents the radiometry from a specific measurement (single sensor, but multiple times). It allows data cleaning, filtering and plotting.
RadiometryGroup: as aforementioned, to obtain the reflectance, it is necessary to use values from 3 different sensors. In this regard, the RadiometryGroup class stores many radiometries and makes easier to apply filters to all of them at the same time. It supports also plotting functions.
RadiometryDB: finally, the last class is responsible for creating and maintaining a radiometry database, from the many measurements that have been treated.
The package is available in the following GitHub repository : https://github.com/cordmaur/RadiometryTrios
The repository contains detailed information about installation and dependencies, but, in summary, it is necessary to just clone the project and install it.
git clone https://github.com/cordmaur/RadiometryTrios.gitcd RadiometryTriospip install -e .
Note: The -e option, will install the package in developer mode, so it makes easier to make changes to the source code (on the project folder) and they will be applied automatically, without necessity to reinstall it.
To check if it is installed correctly, run the following commands on a python shell (or Jupyter notebook):
import RadiometryTrios RadiometryTrios.__version__'0.0.1'
Once we have the package successfully installed, it is time to learn how to use it’s many functionalities.
The first class that will be covered is the TriosMDB class. The notebook 00_TriosMDB.ipynb available at the nbs/ folder describes in detail how to use it.
It is important to note that the TriosMDB class takes care of the ODBC connection under the hood. That means you don't have to take care of it manually and the only requirement is the installation of the pyodbc package that is mandatory. Let’s check how to open a connection and display a summary:
{'FileName': WindowsPath('../mdbs/trios_test.mdb'), 'Records': 937, 'IDDevice': "['IP_c090', 'SAM_83AE', 'SAM_83ba', 'SAMIP_5078', 'SAM_83B0', 'SAM_83BA']", 'IDDataType': "['Inclination', 'Pressure', 'SPECTRUM', 'SAMIP']", 'MethodName': "['SAMIP_5078', 'SAM_83AE', 'SAM_83B0', 'DoubleSpecCalc_2', 'SAM_Calibration_Station', None]", 'IDMethodType': "['SAMIP Control', 'SAM Control', 'DoubleSpecCalc', 'SAM Calibration Station', None]", 'Comment': "['Ed deck', 'Lu deck', 'Ld deck', 'reflectance', None]", 'CommentSub1': "[None, 'factor 0.028']", 'CommentSub2': '[None]'}
Once a TriosMDB object is created, it fills up a pandas DataFrame with all the content from the tblData table. This can be accessed through the .df attribute. The tblData is the default table of the Trios/MSDA format. To access the contents:
mdb.df.head(3)
Probably the most basic action to analyse the MDB file is to perform queries. The function exec_query returns the results of a query as a list or a pandas dataframe. The exec_query can be used to inspect other tables from the MDB file as well. Let's check the contents of the tblData (the main table). The results output can be a dataframe or a list where each item is a tuple that represents a table row.
mdb.exec_query("select top 1 * from tblData", output_format='pandas')
Or the results can be displayed as a list:
results = mdb.exec_query("select top 1 * from tblData", output_format='list')results[0][:6]('2572_2015-11-14_14-29-21_188_038', 1, 'IP_c090', 'Inclination', 'Calibrated', None)
To retrieve the measurements of an MDB file, it is necessary to specify which conditions should be met for the desired radiometry type. These conditions may be a combination of different Columns/Values and must be specified in a dictionary. Each condition’s combination will be related to one radiometry type, also called r-type. For example, in this MDB, we can see that there are reflectances already calculated. Inspecting the MDB in ACCESS, we can see that these reflectances are associated with some IDDevices:
To access the desired radiometry, we need first to create a dictionary with the conditions for each radiometry type, as demonstrated bellow. Once the select_radiometries is successfully called, it will retrieve the times that coincide with all the types, if logic='and' or all times found on the MDB if logic='or'. Each radiometry DataFrame will be loaded in the .measurements attribute. That will be explained in the Accessing Measurements section.
DatetimeIndex(['2015-11-14 14:39:45', '2015-11-14 14:39:55',               '2015-11-14 14:40:05', '2015-11-14 14:40:15',               '2015-11-14 14:40:25', '2015-11-14 14:40:35',               '2015-11-14 14:40:45', '2015-11-14 14:40:55',               '2015-11-14 14:41:05', '2015-11-14 14:41:15',               '2015-11-14 14:41:25', '2015-11-14 14:41:35',               '2015-11-14 14:41:45', '2015-11-14 14:41:55',               '2015-11-14 14:42:05', '2015-11-14 14:42:15',               '2015-11-14 14:42:25', '2015-11-14 14:42:35',               '2015-11-14 14:42:45', '2015-11-14 14:42:55',               '2015-11-14 14:43:05', '2015-11-14 14:43:15',               '2015-11-14 14:43:25', '2015-11-14 14:43:35',               '2015-11-14 14:43:45', '2015-11-14 14:43:55',               '2015-11-14 14:44:05', '2015-11-14 14:44:15',               '2015-11-14 14:44:25', '2015-11-14 14:44:35',               '2015-11-14 14:44:45', '2015-11-14 14:44:55',               '2015-11-14 14:45:05', '2015-11-14 14:45:15',               '2015-11-14 14:45:25', '2015-11-14 14:45:35',               '2015-11-14 14:45:45', '2015-11-14 14:45:55'],              dtype='datetime64[ns]', freq='10S')
The objective of the TriosMDB class is basically search and export the measurements from the Trios/MSDA mdb. To more advanced manipulation, such as interpolation, reflectance calculation or data cleaning, the Radiometry class is more appropriated. Even so, the TriosMDB is filled with some basic plotting functions, to make sure what is being exported makes sense.
mdb.plot_radiometry('reflectance', min_wl=380, max_wl=950)
mdb.plot_radiometries(cols=2,  min_wl=380, max_wl=950)
The last step is to export the radiometries to a text file. The output is a .txt file in the Trios/MSDA format (similar to the .mlb files)The create_measurement_dir flag indicates if the a subdirectory must be created. The subdirectory name follows the format YYYYMMDD-mmss and is the format used by the RadiometryDatabase class.Keep in mind that this format is not easy to be manipulated. It is intended to maintain compatibility with MSDA output files.For a more user-friendly format, it is suggested to use the interpolated output provided by the Radiometry class.
out_dir = mdb.export_txt(create_measurement_dir=True)Saving output file to ..\mdbs\20151114-1439
The output directory is returned. It is a good practice to store it in a variable, because it will be used to open the Radiometry class. Let's now check the files that have been saved:
[file for file in out_dir.iterdir()][WindowsPath('../mdbs/20151114-1439/Ed_interpolated.bak'),  WindowsPath('../mdbs/20151114-1439/Ed_interpolated.csv'),  WindowsPath('../mdbs/20151114-1439/Ed_spectrum_LO.bak'),  WindowsPath('../mdbs/20151114-1439/Ed_spectrum_LO.txt'),  WindowsPath('../mdbs/20151114-1439/Fig_20151114-1439.png'),  WindowsPath('../mdbs/20151114-1439/Ld_interpolated.bak'),  WindowsPath('../mdbs/20151114-1439/Ld_interpolated.csv'),  WindowsPath('../mdbs/20151114-1439/Ld_spectrum_LO.bak'),  WindowsPath('../mdbs/20151114-1439/Ld_spectrum_LO.txt'),  WindowsPath('../mdbs/20151114-1439/Lu_interpolated.bak'),  WindowsPath('../mdbs/20151114-1439/Lu_interpolated.csv'),  WindowsPath('../mdbs/20151114-1439/Lu_spectrum_LO.bak'),  WindowsPath('../mdbs/20151114-1439/Lu_spectrum_LO.txt'),  WindowsPath('../mdbs/20151114-1439/reflectance_spectrum_LO.txt'),  WindowsPath('../mdbs/20151114-1439/Rrs_interpolated.bak'),  WindowsPath('../mdbs/20151114-1439/Rrs_interpolated.csv')]
To check the contents of the reflectance file. We can open it in the Notepad or any other text editor:
As already mentioned, we can see that this format is not user friendly to be opened in Excel or in a Pandas DataFrame, for example. The interpolated .csv output provided by the Radiometry class is more appropriate for these purposes.
All the code, filled with more examples, is available at the /nbs/00_TriosMDB.ipynb notebook:
In this first part, we’ve seen how to use the TriosMDB class to export radiometry measurements from the .mdb files at the MSDA/Trios format. Basic filtering functionalities are provided to locate the exact measurement inside the .mdb file and export it to a text file. More advanced manipulation functions, such as data cleaning, filtering, interpolation, etc. are provided by the Radiometry and the RadiometryGroup classes, but they will be covered in the next part.
Thanks, and see you there.
If you liked this article and want to continue reading/learning without limits, consider becoming a Medium member. I’ll receive a portion of your membership fee if you use the following link, with no extra cost to you.