Package: readgssi
readgssiCopyleft 🄯 2017-2022This program was written to read and process ground-penetrating radar files from instruments made by Geophysical Survey Systems Incorporated (GSSI), although I have no affiliation with nor endorsement for the afforementioned organization.The demands of ice- and ground-penetrating radar (GPR) surveying, as in many types of scientific fieldwork, require that both quality control and time savings are critical to a successful field campaign. This software provides a way to quickly read, process, and display radar data produced by GSSI radar antennas and control units.readgssiwas designed to be used in the field to quality-check entire folders of data files by converting radar profiles to portable network graphics (PNG) images, saving users valuable time versus performing the equivalent actions by hand in RADAN, especially in the case of projects with large file counts.readgssihas the capability to read DZT and DZG files with the same pre-extension name and plot the data contained in those files.readgssiis also currently able to translate most DZT files to CSV andnumpybinary (seefuture). The file read parameters are based on GSSI's DZT file description, similar to the ones available on pages 55-57 of theSIR-3000 manual. File structure is, unfortunately, prone to change at any time, and although I've been able to test with files from several systems, I have not encountered every iteration of file header yet. If you run into trouble, pleasecreate a github issue.Full documentation, including a tutorial, is available athttps://readgssi.readthedocs.io.Questions, feature requests, and bugs: pleaseopen a github issue. Kindly provide the error output, describe what you are attempting to do, and attach the DZT/DZG file(s) causing you trouble.requirementsStrongly recommended to install viaanaconda:obspymatplotlibnumpyscipypandash5pyInstall viapip:pynmea2geopypytzinstallationOnce you haveanacondarunning, installing is pretty easy.You can install via theanaconda cloud:condacreate-nreadgssi-ciannesbittreadgssi# and then activate by doing:condaactivatereadgssior viapip/PyPI:condaconfig--addchannelsconda-forge
condacreate-nreadgssipython==3.7pandash5pypytzobspy
condaactivatereadgssi
pipinstallreadgssiThat should allow you to run the commands below.installing from source:If you choose to install a specific commit rather than the latest working release of this software, you may download this package, unzip to your home folder, open a command line, then install in the following way:pipinstall~/readgssiusageTo display the help text:$readgssi-h

usage:
readgssi-iinput.DZT[OPTIONS]requiredflags:OPTION|ARGUMENT|FUNCTIONALITY
-i,--input|file:/dir/f.DZT|inputDZTfile

optionalflags:OPTION|ARGUMENT|FUNCTIONALITY
-o,--output|file:/dir/f.ext|outputfile.ifnotset,willbenamedsimilartoinput
-f,--format|string,eg."csv"|outputformat(CSVandDZTaretheonlyworkingformatscurrentlyavailablefromthecommandline)-p,--plot|+integeror"auto"|plotsize.willbexincheshighor"auto".default:10.seealso-DtosetDPI
-D,--dpi|positiveinteger|settheplotDPIforfiguremaking.defaultsto150-T,--titleoff||turntheplottitleoff(usefulforfiguremaking)-x,--xscale|string,eg."dist"|xunits.willattempttoconvertthex-axistodistance,time,ortraceunitsbasedonheadervalues
-z,--zscale|string,eg."time"|zunits.attempttoconvertthex-axistodepth,time,orsampleunitsbasedonheadervalues
-e,--zoom|listof+int[LRUD]|setazoomtoautomaticallyjumpto.listorderis[left,right,up,down]andunitsarethesameasaxis
-n,--noshow||suppressmatplotlibpopupwindowandsimplysaveafigure(usefulformulti-fileprocessing)-c,--colormap|string,eg."Greys"|specifythecolormap(https://matplotlib.org/users/colormaps.html#grayscale-conversion)-g,--gain|positivefloat|gainconstant(higher=greatercontrast,default:1)-A,--absval||Displaystheabsolutevalueoftheverticalgradientofthearraywhenplotting.Goodfordisplayingfaintarrayfeatures.
-r,--bgr|+integerorzero|horizontalbackgroundremoval(usefultoremoveringing).zero=fullwidth;positive=windowsize(afterstacking)-R,--reverse||reverse(fliparrayhorizontally)-w,--dewow||trinomialdewowalgorithm
-t,--bandpass|+int-+int(MHz)|triangularFIRbandpassfilterappliedvertically(positiveintegerrangeinmegahertz;ex.70-130)-b,--colorbar||addacolorbartotheradarfigure
-a,--antfreq|positiveinteger|setantennafrequency.overridesheadervalue
-s,--stack|+integeror"auto"|settracestackingvalueor"auto"toautostackto~2.5:1x:yaxisratio
-N,--normalize||distancenormalize;reads.DZGNMEAdatafileifitexists;otherwisetriestoreadCSVwithlat,lon,andtimefields
-P,--pausecorr||pausecorrection;fixesdecouplingofDZGandDZTtracenumbersduringsurveypausesusinglowvelocityGPSmarks
-d,--spm|positivefloat|specifythesamplespermeter(spm).overridesheadervalue
-m,--histogram||produceahistogramofdatavalues
-E,--epsr|float>1.0|user-definedepsilonsubr(sometimesreferredtoas"dielectric")ifset,ignoresvalueinDZTheader
-Z,--zero|+intorlistofint|timezero:skipsamplesbeforedirectwave.samplesareremovedfromthetopofthetrace.uselistformulti-channel

namingschemeforexports:CHARACTERS|MEANINGCh0|Profilefromchannel0(canrangefrom0-3)Dn|DistancenormalizationTz233|Timezeroat233samplesS8|Stacked8timesRv|Profilereadinreverse(flippedhorizontally)Bgr75|Backgroundremovalfilterwithwindowsizeof75Dw|DewowfilterBp70-130|triangularFIRfilterappliedfrom70to130MHzG30|30xcontrastgainAbs|ColorscalerepresentsabsolutevalueofverticalgradientZ10.20.7.5|zoomfrom10-20axisunitsonthex-axisand5-7onthez-axisFrom a unix command line:readgssi-iDZT__001.DZTSimply specifying an input DZT file like in the above command (-i file) will display a host of data about the file including:name of GSSI control unitantenna modelantenna frequencysamples per tracebits per sampletraces per secondL1 dielectric as entered during surveysampling depthspeed of wave at given dielectricnumber of tracesnumber of secondsbasic functionalityCSV outputreadgssi-iDZT__001.DZT-otest.csv-fCSVTranslates radar data array to CSV format, if that's your cup of tea. One might use this to export to Matlab. One CSV will be written per channel. The script will rename the output to 'test_100MHz.csv' automatically. No header information is included in the CSV.readgssi-iDZT__001.DZT-s8-r0-otest.csv-fCSVApplies 8x stacking, and background removal filters before exporting to CSV (see explanations below).DZT output (experimental)readgssi-iDZT__001.DZT-R-Z0-otest.DZT-fDZTThis exports the array as a second DZT file after reversing the direction of the survey line.The following operations will stick when writing files:stackingfilteringdistance normalizationreversing directionzero (top of array will be shaved off; to avoid this, set-Z 0. this will be fixed in the future.)manually setting header values, ror example samples per meter (-d 10), epsilon_r (-E 3.1), and antenna frequency (-a 100)The following operations willnotstick:absolute value of gradientgainzoomplot features like axis units (obviously)Please note the following about DZT output:While DZTs will be written, there will be no accompanying DZX written. This functionality may be implemented in the future but for now is a bit too complex of an operation for readgssi. This could have consequences if you're planning to re-import to RADAN after this operation.Single channel files onlyare supported at this time. Multi-channel files will be written but the values on the second channel will not be correct. If you wish to use readgssi to re-write multiple channels, you will have to separate the channels using RADAN or another software first.numpy object outputreadgssi-iDZT__001.DZT-otest.npy-fnumpyThis command will save the array in numpy binary format. Header information will not be saved, however.GPRPy-compatible outputreadgssi-iDZT__001.DZT-otest-fgprpyThis command saves the array in numpy binary format, and preserves the header as a JSON file as well.GPRPywill soon support importing this type of file.plottingexample 1A: without gainreadgssi-iDZT__001.DZT-o1a.png-p5-sautoThe above command will causereadgssito save and show a plot named "DZT__001c0Tz233S6G1.png" with a y-size of 5 inches at 150 dpi (-p 5) and the autostacking algorithm will stack the x-axis to some multiple of times shorter than the original data array for optimal viewing on a monitor, approximately 2.5*y (-s auto). The plot will be rendered in thegraycolor scheme.example 1B: with gainreadgssi-iDZT__001.DZT-o1b.png-p5-sauto-g60-r75This will causereadgssito create a plot from the same file, but matplotlib will save the plot as "1b.png" (-o 1b.png). The script will plot the y-axis size (-p 5) and automatically stack the x-axis to (-s auto). The script will plot the data with a gain value of 50 (-g 50), which will increase the plot contrast by a factor of 50. Nextreadgssiwill run the background removal (-r 75) filter. (To apply this horizontal background removal filter across the entire line horizontally, use-r 0. More about this in examples2band2c.)example 1C: display the vertical axis in meters and manually set the Epsilon_rreadgssi-iDZT__001.DZT-o1c.png-p5-sauto-g60-r75-zm-E80This will set the Epsilon_r—that is, the relative permittivity of the first layer medium—to 80 (-E 80) and the Z axis to meters (-z m). The resulting plot will have a depth scale in meters.
Epsilon_r should be set in the header, but if it is set incorrectly, this is the way to adjust it on the fly.example 1D: make a high-quality figurereadgssi-iDZT__001.DZT-o1d.png-p5-sauto-g60-r75-zm-E80-T-D300Okay, now our figure looks great and we want to put it in a publication! Time to turn off the title (-T) and pump up the DPI to printer quality (-D 300). For AGU poster-quality images, you may even need-D 600.
One tricky thing to keep in mind is that the smaller the plot size (-p) the larger the relative size of the text in the figure.example 1E: changing colormaps and gain settingsreadgssi-iDZT__001.DZT-o1e.png-p5-sauto-g20-r75-zm-E80-cseismicHere, a horizontal background removal is applied, but gain is turned down (-g 20). The script uses matplotlib's "seismic" colormap (-c seismic) which is specifically designed for this type of waterfall array plotting. Even without gain, you will often be able to easily see very slight signal perturbations. Given its use of red, however, it is not terribly colorblind-friendly for either of the two most common types of human colorblindness, which is why it is not used as the default colormap.example 1F: absolute value of vertical gradientreadgssi-iDZT__001.DZT-o1f.png-p5-sauto-g100-r75-zm-E80-cgray_r-A-t80-120While we're on the topic of colormaps, it's worth pointing out that you can tell readgssi to calculate the vertical derivative (the "gradient") of the profile and display its absolute value using the-Aflag. This gradient display is a good way to highlight areas of polarity change regardless of positive or negative values. It is particularly useful for highlighting glacial bed material through ice, for example. Here (in a lake profile) we set both-Ato plot the absolute value of vertical gradient and also a FIR filter (-t 80-120) explainedbelow. We also set the colormap to the reverse of the usual one (-c gray_r) so that black indicates high values.filteringEach of the plots in this section is displayed with a histogram to illustrate changing array values.example 2A: no background removalreadgssi-iDZT__002.DZT-o2a.png-p5-sauto-g30-mSometimes, files will look "washed out" due to a skew relative to the mean of the data. This is easily correctable. Herereadgssiwill create a plot of size 5 and stack 5x (-p 5 -s 5). Matplotlib will use the default "gray" colormap and save a PNG of the figure, but the script will suppress the matplotlib window (using the-nflag, useful for processing an entire directory full of DZTs at once). Finally, the-mflag will draw a histogram for each data channel. Note how the histogram changes when filters are applied.example 2B: horizontal mean BGR algorithm appliedThe flag to get rid of the skew (or any horizontally uniform noise) is-r, also known as background removal or BGR for short.-rhas two modes, one set by-r 0and one set when the option after the-rflag is greater than zero. When this BGR option is zero, the program simply subtracts the average of each profile row from the array. When it's greater than 0,readgssiwill implement a moving window mean, the size of which is set inpost-stacktraces (seeexample 2C).The command below does the same thing asexample 2A, except-r 0applies full width horizontal mean background removal to the profile. Note the difference in ringing artifacts and skew between examples 2a and 2b.readgssi-iDZT__002.DZT-o2b.png-p5-sauto-g30-m-r0example 2C: moving window horizontal meanreadgssi-iDZT__002.DZT-o2c.png-p5-sauto-g30-m-r75Same as above but with a 75-trace wide moving window mean (-r 75). This width represents post-stack traces. This is, for all intents and purposes, the same as RADAN's "BOXCAR" method of horizontal noise removal, but much, much faster because it'simplemented with scipy. Areas beyond the left and right edges are treated as zeros. Notice that the noise in the water column is nearly entirely wiped out, but real data is extended with lateral wisps the size of half of the window, which is a side-effect of this method. Note that the histogram (-m) has a fairly even distribution around the mean, which generally indicates that the image should be fairly readable.example 2D: vertical triangular FIR filterreadgssi-iDZT__002.DZT-o2d.png-p5-sauto-g30-m-t80-120This function is functionally the same as RADAN's vertical triangular FIR filter, but againimplemented with scipy. It is used to filter out unwanted frequencies on a per-shot basis, which makes it very effective as a profile cleaning tool. The most effective results are achieved by filtering a tight window of frequencies around the center frequency of the antenna you are using. In this case, we are using a 100 MHz antenna and filtering from 80-120 MHz.example 2E: combining filters for a cleaner profilereadgssi-iDZT__002.DZT-o2e.png-p5-sauto-g30-m-r75-t80-120In this example, we use two concurrent filters (the horizontal windowed BGR and the vertical triangular FIR) to try and clean the image up as much as possible without overcooking it.array manipulationexample 3A: (without) distance normalizationThe default behavior ofreadgssiis to plot the X-axis in survey time units (seconds). This can be changed using the-xflag. To display in distance units, you must either have GPS information in DZG format, or specify the number of radar traces per meter using the-dflag.-d 24 -x meterswill change the traces per meter value in the header to 24.0 and display the profile with distance in meters along the X-axis.Files with GPS information are handled in a slightly different way. First,readgssiwill read a DZG file to create an array of distance information associated with marks in the DZT.(NOTE: If your project was recorded without DZG files but you have per-line GPS mark information in GPX format, please look atgpx2dzgfor a method of creating a DZG file for each survey line)After reading the DZG, the program will expand or contract the GPR array based on the speed over ground between GPS points. It will then modify the traces per meter value from the header and display the profile with distance on the X-axis.Here a file is processed and displayed without distance normalization:readgssi-iDZT__003.DZT-o3a.png-p5-sauto-r0-g60example 3B: distance normalization using a DZG fileTo use DZG GPS information to distance normalize the profile and display in meters traveled, use the-Nand-x metersflags.readgssiwill normalize the file in chunks to reduce memory usage. Here is the same file with distance normalization applied:readgssi-iDZT__003.DZT-o3b.png-p5-sauto-r0-g60-N-xmetersexample 3c: reverse line directionA simple but sometimes necessary operation is flipping a survey line back to front in order to display it congruently with others in a set. This can be done easily using-R.advanced usageUNIX users have a distinct advantage of being able to easily process entire folders full of DZTs with a simple command. Users who wish to do this shouldread upon how to constructforloops in Bash or simply follow and modify these examples below.processing all files in a folderThis command makes use of thelsfunction in Bash, which lists all files that match a specific pattern. In this case, we want the pattern to be "any DZT file," which ends up being simplyls *.DZT(the*symbol is a wildcard, meaning it matches any set of characters, so in this case it would match bothFILE____005.DZTandtest.DZTbut notTest01.dztbecause the.DZTis case sensitive.).forfin`ls*.DZT`;doreadgssi-p8-n-r0-g40-Z233-zns-N-xm-sauto-i$f;doneThe structure of this command is easy to understand if you know a little bit aboutforloops. This command loops over every file with the extension.DZT(ls *.DZTwhere*indicates a wildcard) and assigns the filename to thefvariable on each loop. Then, after the semicolon, bash runs readgssi for every pass of the loop. In this case, the parameters are:-p8# plot with size 8-n# suppress the matplotlib window; useful if you do not want the operation interrupted by plot windows-r0# full-width background removal-g40# gain of 40-Z233# time zero at 233 samples-zns# display the depth axis in nanoseconds-N# distance-normalize the profile-xm# display the x-axis in meters-sauto# apply automatic stacking-i$f# recall the `f` variable containing this loop's filename and feed it to the input flag of readgssiFinally, end the loop by closing the command with a linebreak;, and thedonemarker.processing specific sets of filesYou can make the command even more specific by further modifying the set of files returned by thelscommand. For example:forfin`lsFILE__{010..025}.DZT`;doreadgssi-p8-n-r0-g40-Z233-zns-N-xm-sauto-i$f;doneThis command will process only the 16 files in the numeric sequence between and including010and025in the set (FILE__010.DZT,FILE__011.DZT,...,FILE__025.DZT).bashhandles the zero padding for you as well. Pretty cool.contributorsIan Nesbitt (@iannesbitt, author)François-Xavier Simon (@fxsimon)Thomas Paulin (@thomaspaulin)teshaw (@teshaw)citation suggestion:Ian M. Nesbitt, François-Xavier Simon, Thomas Paulin, 2018. readgssi - an open-source tool to read and plot GSSI ground-penetrating radar data.doi:10.5281/zenodo.1439119known bugs:scipy 1.2.x causes errors when filtering. use scipy 1.3.0 to avoid.futureautomatic script testing for smoother devcreate a class for surveyline objects, similar toobspy.core.trace.TraceGPS transcription from CSV with fields likemark name, lon, lat, elev, timeability to use GPS altitude to adjust z position across profileability to ground truth depths as horizontal linesGUI-based geologic/dielectric layer pickinglayer velocity calculation (using minimum of clustered hyperbola tail angle measurements, or manual input)velocity-based depth adjustmentsability to incorporate ground truth measurementsvelocity gradient/angle of incidence-based array migrationtranslation to common geophysical formats (HDF5, SEGY, etc.)integration withvistafor 3D visualization of location-aware arrays