Package: raadpy
raadpy | RAAD Data Analysis FrameworkThis is a python package with the necessary libraries to conduct an analysis on the raw data obtained from the RAAD detector onboard the Light-1 cubesat mission.Table of contentsraadpy | RAAD Data Analysis FrameworkTable of contentsFeaturesInstallationInstalling withpipBuilding from SourceDownload the codeMake sure setuptools are up-to-dateBuild and install theraadpyBasic UsageLoad arrays of events from filenamesPlot events on interactive mapsAutomatically obtain lighting strikes near eventsOne-line reading of the Light-1 payload buffersEasy timestamp correctionFeaturesHere are some of the things you can do withraadpy. Clicking the link will take you to tutorials on how to do any of these tasks.Load arrays of events from filenames:raadpycan load different types of ecents, from lightning strikes to TGF events, to locations of the satellite, etc. Basically there is built-in support for everything that has longitude, latitude, and a timestamp. These arrays have extra features, such as automatic precision in storing timestamps and easy to use conversion between timestamp formats.Plot events on interactive maps: After loading types of events one can plot them on interactive globes that can be exported as animations, interactive html files, or simply publication quality plotsAutomatically obtain lighting strikes near events: Given a set of events (such as TGF events)raadpycan automatically detect nearby lightnings and download them in a python friendly format for computation.One-line reading of the Light-1 payload buffers: the package can be used to decode the binary files from the buffers with 1 line of python code.Easy timestamp correction: We all know what happened with the timestamps and the PPS signal.raadpyoffers a simple way to estimate the timestamp using the order of the data in the payload buffers.These are only some things that the library can do, for a full list of the functions and capabilities please look at thesource code.Installationraadpyis a library build forpython>=3.6however tested onpython>=3.9, we recommend updating to the latest python distribution before installing the code. We further recommend to clonethis python environmentto download additional packages that would be useful. This step is not necessary however.To installraadpythere are two options, installing the latest release throughpip, or installing from source.Installing withpipTo install thorughPyPIusingpipopen a terminal and run$ pip install raadpyThis should install the latest version ofraadpyautomatically. In case this doesn't work, fear not! You can install the library from source.Building from SourceTo buildraadpyfrom source the following tools are needed:Git: Is a tool to download this code to your computer from the terminal. Link points to an installation tutorial.pip: This is PyPI's package manager.If the above is installed on your system then proceed with the following:Download the codeOpen a terminal (Linux and MacOS), or a Powershell (Windows), and use thecdandls(dirin Windows) commands to navigate to the folder you want to downloadraadpy's source code. You can download it by running the following$ git clone https://github.com/nyuad-astroparticle/raadpy.git
$ cd raadpy/After that you should have a directory calledraadpythat contains the contents of this repo, and with the second command you should be in it.Make sure setuptools are up-to-dateTo ensure that the tools you need to build the library are up to date run this next$ python3 -m pip install --upgrade pip setuptools wheel buildBuild and install theraadpyNext we will buildraadpyby running$ python3 -m buildand we will install by running$ pip install .The "." at the end is important as it specifies that we want to install whateevr is in the current directory. Andvoila!You should haveraadpysuccessfully installed.Basic UsageThis is a short tutorial to useraadpy. It is meant as a short guide to understand how the package is structured and what are the main modules one can work with.Load arrays of events from filenamesWithraadpyyou can load events and convert them to a powerful python friendly format simply by using the filename. In this example we will load a set of TGF's from tehFermimission and then we will examine what we can do with this dataset.You can do this in a jupyer notebook or in a simple python script. To load the data we will first importraadpyand then do the following# Import libraryimportraadpyasrp# Define a variable with the filename where the FERMI data is storedfermi_filename="PATH-TO/FERMI-data.txt"# Replace this path with yours# Load the FERMI datadata=rp.array(filename=fermi_filename,event_type='fermi')And that's it! Now you have created araadpy arraythat contains the data from this filename specially formatted asFermimission data.Araadpy arraycan be thought of a list, it's structure is similar to anumpy array, so you can still access an event by doingdata[3], and perform all the other methods familliar to lists such aslen(data),for datum in data, etc. However this array has a special constructor that allows it to automatically format certain types of event. You can specify the type of event using theevent_typeargument as shown above. The followingevent_typesare supportedlocation:(default)Simply holds a location and timestamp of events over a map.fermi: TGFs from the fermi missionlight-1: Events from theLight-1missionlightning: Lightnings usually downloaded fromblitzortung.org.You can set up your array to be any of these types, just know thatevent_typefield is case sensitive, so enteringLIght-1might result in an error.But what can you do with the events after they are loaded? Well a bunch of things! First letsprinta snapshot of them in order to examine them. This can simply be done by running# Print the dataprint(data)this produces the follwoing output0TGF:TGF210628068|Mission:fermi
Timestamp(ISO):2021-06-2801:37:54.440
Lat:13.0000Long:-87.6833
Detector_id:1000100000001TGF:TGF210627854|Mission:fermi
Timestamp(ISO):2021-06-2720:30:00.815
Lat:22.6000Long:-100.2500
Detector_id:1010012TGF:TGF210620681|Mission:fermi
Timestamp(ISO):2021-06-2016:20:45.079
Lat:-3.0167Long:144.0500
Detector_id:100100000000103TGF:TGF210617554|Mission:fermi
Timestamp(ISO):2021-06-1713:17:19.041
Lat:20.2667Long:78.1833
Detector_id:114TGF:TGF210617308|Mission:fermi
Timestamp(ISO):2021-06-1707:24:08.879
Lat:10.2500Long:-84.0333
Detector_id:11...
Lat:2.7830Long:-69.9670
Detector_id:11As you can see the pkey of the event is printed, followed by it's unique ID, the mission time, the timestamp in ISO, then the latitude and longitude, and finally the id of the detector the event was from.Let's start using the library now, we can obtain a list of the timestamps of all of the envets in any time format like so:# Get the timestamps in mjd formattimestamps_mjd=data.get_timestamps(format='mjd')print(timestamps_mjd)the output is the followingarray([59393.0679912, 59392.8541761, 59385.6810773, 59382.5536926,
       59382.3084361, 59379.7080586, 59371.2641016, 59354.7158131,
       59349.5049853, 59348.0465583, 59347.394978 , 59345.8690265,
       59339.3054766, 59338.8064969, 59338.3140316, 59337.4563384,
       59332.6334789, 59331.3091693, 59330.9530484, 59330.4564704,
...
       55635.6609862, 55632.1763637, 55630.7547783, 55629.2279641,
       55614.4684369, 55610.83396  , 55604.8978094], dtype=float64)This actually offers arbitrary conversions to well known timestamps, here we chosemjdbut you can use anything fromunix,iso, and more! A numpy array is returned for easy further manipulation.Similarly one can obtain tuples with the latitude and logitude of the elements of the arrays like so:# Get the coordinates of the events in lon-latpositions=data.get_coords()print(positions)The output is as followsarray([[ -87.6833,   13.    ],
       [-100.25  ,   22.6   ],
       [ 144.05  ,   -3.0167],
       ...,
       [ 143.817 ,  -20.833 ],
       [ 100.667 ,    3.667 ],
       [ -69.967 ,    2.783 ]])still a numpy array is returned with the appropriate entries as described above.Plot events on interactive mapsWithraadpyyou can plotarraysin interactive maps usingplotly. In this example we will load the path of the cubesat and plot it on an interactive javascript map.First we load the data# Import the libraryimportraadpyasrp# Filename of datapath_filename='PATH-TO/LIGHT-1_LOCATIONS.csv'# Change the path accordingly# Create raadpy arraydata=rp.array(filename=path_filename,event_type='locations')# Plot the first 10000 eventspath=rp.array(data[:10000])# Get the first 10000 eventsrp.map(path,long=-80,lat=20)# Plot themThe output looks like thisBut the only difference is that in the python environment this is actually a 3D model that you can move, export into an image, html interactive file, or even an animation!Automatically obtain lighting strikes near eventsTo automatically obtain lightning strikes near events one can simply load a set of events, just like theFermiTGFs loaded in the first example, and with a couple more commands the nearest lightnings will show up. This is at the code below# Pick a TFG from the data in the first exampleTGF=data[10]# Find all the ligtnings within 10s of itlights=rp.download_lightnings(TGF.timestamp,threshold=10)# Filter those lightnings to find the ones that are closest in time from the TGFnear=rp.get_nearby_lightning(TGF,lightnings=lights,threshold=1)print(near)# Finally plot the outcome on the maprp.map(TGF,near,name1='TGF',name2='Lightning')The output should look something like thisWarning! Threshold: 10.000000 s, is too small to be detected by Blitzortung! Using threshold = 6 * 60 s instead.
Searching for Lightnings between:
	 start-time: 2021-06-28 01:31:54.440
	 end-time:   2021-06-28 01:43:54.440
Found Lightning data at: https://www.blitzortung.org/en/archive_data.php?stations_users=0&selected_numbers=*&end_date=1624838400&end_time=6234&start_date=1624838400&start_time=5514&rawdata_image=0&north=90&west=-180&east=180&south=-90&map=0&width_orig=640&width_result=640&agespan=60&frames=12&delay=100&last_delay=1000&show_result=1
Data Downloaded Successfully
    0 Lightning: -79.891498 | Mission: Blitzurtong
        Timestamp (ISO): 2021-06-28 01:37:54.938
        Lat:   14.9226 	 Long:  -79.8915
        Detector_id: Blitz
        
    1 Lightning: -90.06834 | Mission: Blitzurtong
        Timestamp (ISO): 2021-06-28 01:37:53.891
        Lat:   33.1204 	 Long:  -90.0683
        Detector_id: Blitz
        
    2 Lightning: -90.026664 | Mission: Blitzurtong
        Timestamp (ISO): 2021-06-28 01:37:53.892
        Lat:   33.2297 	 Long:  -90.0267
        Detector_id: Blitz
...The warning is not a big deal as it is simply saying that the website the data is downloaded from can only get the lightnings at intervals of 5 minutes. Therefore it is rebsing the search of lightnings to a 5 minute interval between them. Then the nearest lightnings are printed, and finally an interactive map of the lightnings is shown.One-line reading of the Light-1 payload buffersPerhaps one of the most useful features of theraadpylibrary is that it can read the buffers of the Light-1 Payload with one-line commands, as well as filter the events accordingly.In this example we load the data for the orbit buffer and filter for the PMT signal only, then plot.# Import the libraryimportraadpyasrp# Define the filename of the datafilename='PATH-TO-ORBIT/data.dat'# Change this for your data# Load the orbit buffer and apply a filter for the PMT data onlydata=rp.get_dict(filename,struct=rp.ORBIT_STRUCT,condition="data['id_bit'] == 1")# Plot the decoded datafig,axes=rp.plot_buffer(data)Note that theget_dictfunction returns a dictionary. Each entry of the dictionary is a numpy array of all the measurements on the file, and the entries of the dictionary are decided by predifined constants, in this caserp.ORBIT_STRUCT. As a resutl, to read a buffer, one has to specify the structure of the file in a dictionary using thestructargument. The dictionary looks like the example shown below for the orbit buffer.ORBIT_STRUCT={'timestamp':32,'temperature':8,'rate0':12,'rate1':12,'rate2':12,'rate3':12,'ratev':8,'hv_level':12,'veto_level':12,'id_bit':1,'pps_active':1,'suspended':1,'power_on':1,'scenario':4,}The library however, has three structures predefined these are as follows so that the user doesn't have to do it from scratch:Orbit Buffer:rp.ORBIT_STRUCTVeto Buffer:rp.VETO_STRUCTNon-Veto Buffer:rp.NONVETO_STRUCTSo these can be used directly out of the box. An in-depth application of reading all the buffers is shownhere.The output of the previous code looks like thisNOTE:There are two different ways that the buffers are encoded. If one wants to decode the Buffers for the VETO and NONVETO they must includeSTUPID=Trueas an argument onrp.get_dict()to specify the decoding method.Easy timestamp correctionAs we know the corruption of the timestamp signal poses a major issue for the analysis of the Light-1 data. The algorithm derived to correct the timestamps is included in the library and can be invoked on a set oforbitdata andbufferdata like so# Filter events and correct timestamp based on orbit rate and buffer ordertimestamp,total_cnt,valid_events=rp.correct_time(buffer,orbit,TIME,RANGE_ORBIT,RIZE_TIME,CONST_TIME,TMAX)The arguments torp.correct_time()are as follows:buffer: The decoded buffer that contains the events we are interested in correting their timestamporbit: The equivalent orbit bufferTIME: The Period of the orbit data collection in seconds (normally set to 20s)RANGE_ORBIT: A tuple of indices that mark the beggining and end of the orbit data we want to look at (e.g. (300,5000))RIZE_TIME: The time it takes for the FPGA counter to reach saturationCONST_TIME: The time it the FPGA counter is saturated per cycleTMAX: The maximum number reached by the FPGA (999 for VETO and 9999 for NONVETO)An example of using this function to correct the timestamp is shownhere.