Package: nais-processor
NAIS ProcessorCode package to processNAIS(Neutral cluster and Air Ion Spectrometer, Airel Ltd.) data files.Installationpipinstallnais-processorYou can find the package onPyPI.DocumentationSeehereModulesProcessorThenais.processormodule can be used to process the data to netcdf files and allows options for the following operations:Inlet loss correction (Gromley and Kennedy, 1948)Ion mode correction (Wagner et al. 2016)Conversion to standard conditions (293.15 K, 101325 Pa)Remove charger ion band from total particle dataUse fill values in case of missing environmental sensor dataUtilsThenais.utilsmodule contains functions that allow one to do operations on the NAIS data files.CheckerThenais.checkermodule contains a GUI application with which one can visually inspect the nais ion/aerosol size distributions along with the flags and identify bad data by drawing a bounding box around it and saving the coordinates for later use.(Tested with Qt vers. 5.15.2)Example usageUse themake_config_template()method to create a configuration file template and fill it with necessary information. The configuration file is used at processing the data files.fromnais.processorimportmake_config_templatemake_config_template("/home/user/viikki.yml")Running this will create a configuration file template called/home/user/viikki.yml. After filling in the information in the confguration file for our example measurement the file may look like this:measurement_location:Viikki, Helsinki, Finlanddescription:Agricultural sitelongitude:25.02latitude:60.23data_folder:-/home/user/data/2021-/home/user/data/2022processed_folder:/home/user/viikkidatabase_file:/home/user/viikki.jsonstart_date:2022-09-28end_date:2022-09-30inlet_length:1.0do_inlet_loss_correction:trueconvert_to_standard_conditions:truedo_wagner_ion_mode_correction:trueremove_corona_ions:trueallow_reprocess:falseredo_database:falsefill_temperature:273.15fill_pressure:101325.0fill_flowrate:54.0dilution_on:falsefile_format:blockresolution:5minThen process the data files by runningnais_processor()method with the config file as the input argument.In our example case:fromnais.processorimportnais_processornais_processor("/home/user/viikki.yml")Building database...
Processing 20220928 (Viikki, Helsinki, Finland)
Processing 20220929 (Viikki, Helsinki, Finland)
Processing 20220930 (Viikki, Helsinki, Finland)
Done!The code produces daily processed data filesNAIS_yyyymmdd.nc(netCDF format). These files are saved in the destination given in the configuration file.The locations of raw and processed files for each day are written in the JSON formatteddatabase_file. This prevents reprocessing whenallow_reprocess: false.The netcdf files have the following structure:FieldsDimensionsData typeUnitsCommentsCoordinatestimetimedatetime64[ns]timezone: utcdiameterdiameterfloatmparticle diameterflagflagstringData variablesneg_ionstime,diameterfloatcm-3dN/dlogDppos_ionstime,diameterfloatcm-3dN/dlogDpneg_particlestime,diameterfloatcm-3dN/dlogDppos_particlestime,diameterfloatcm-3dN/dlogDpneg_ion_flagstime,flagintflag=1, no flag=0pos_ion_flagstime,flagintflag=1, no flag=0neg_particle_flagstime,flagintflag=1, no flag=0pos_particle_flagstime,flagintflag=1, no flag=0AttributesMeasurement infodictionaryNext we combine the previously created files into a single continuous dataset with 1 hour time resolution and only raise a flag if at least 50% of the data points inside the two hour window contain the flag. We save it as a netcdf file.fromnais.utilsimportcombine_dataimportpandasaspdimportxarrayasxrdata_source="/home/user/viikki"date_range=pd.date_range("2022-09-28","2022-09-30")ds=combine_data(data_source,date_range,"1H",flag_sensitivity=0.5)ds.to_netcdf("combined_nais_dataset.nc")Then we launch the data checker with the combined data in order to identify bad data. Bounding boxes can be drawn around bad data in the size distributions (initiate an adjustable box with double left click and remove from the menu opened by right clicking the box). By clicking the save boundaries button the box coordinates are saved to a netcdf file (filename given in the second argument). If the bounding boxes are saved, they will be reloaded when the checker is reopened with same arguments, so save your work regularly in case the program crashes.fromnais.checkerimportstartNaisCheckerstartNaisChecker("combined_nais_dataset.nc","bad_data_bounds.nc")We can set the bad data toNaNin our combined file and use the resulting dataset as the starting point for further analysis.fromnais.utilsimportremove_bad_datads=xr.open_dataset("combined_nais_dataset.nc")bad_data=xr.open_dataset("bad_data_bounds.nc")ds=remove_bad_data(ds,bad_data)LicenseThis project is licensed under the terms of the GNU GPLv3.ReferencesGormley P. G. and Kennedy M., Diffusion from a Stream Flowing through a Cylindrical Tube, Proceedings of the Royal Irish Academy. Section A: Mathematical and Physical Sciences, 52, (1948-1950), pp. 163-169.Wagner R., Manninen H.E., Franchin A., Lehtipalo K., Mirme S., Steiner G., Petäjä T. and Kulmala M., On the accuracy of ion measurements using a Neutral cluster and Air Ion Spectrometer, Boreal Environment Research, 21, (2016), pp. 230–241.