Package: shepherd-data
Data ModuleDocumentation:https://orgua.github.io/shepherd/external/shepherd_data.htmlSource Code:https://github.com/orgua/shepherd-datalibMain Project:https://github.com/orgua/shepherdThis Python Module eases the handling of hdf5-recordings used by theshepherd-testbed. Users can read, validate and create files and also extract, down-sample and plot information.InstallationPIP - Onlinepip3installshepherd-data-UFor bleeding-edge-features or dev-work it is possible to install directly from GitHub-Sources (heredev-branch):pipinstallgit+https://github.com/orgua/shepherd-datalib.git@dev#subdirectory=shepherd_data-UProgramming InterfaceBasic Usage (recommendation)importshepherd_dataassdwithsd.Reader("./hrv_sawtooth_1h.h5")asdb:print(f"Mode:{db.get_mode()}")print(f"Window:{db.get_window_samples()}")print(f"Config:{db.get_config()}")Available FunctionalityReader()file can be checked for plausibility and validity (is_valid())internal structure of h5file (get_metadata()orsave_metadata()... to yaml) with lots of additional dataaccess data and various converters, calculatorsread_buffers()-> generator that provides one buffer per call, can be configured on first callget_calibration_data()get_windows_samples()get_mode()get_config()direct access to root h5-structure viareader['element']converters for raw / physical units:si_to_raw()&raw_to_si()energy()sums up recorded power over timedownsample()(if needed) visualize recording (plot_to_file())Writer()inherits all functionality from Readerappend_iv_data_raw()append_iv_data_si()set_config()set_windows_samples()IVonne Readerconvert_2_ivcurves()converts ivonne-recording into a shepherd ivcurveupsample_2_isc_voc()TODO: for now a upsampled but unusable version of samples of short-circuit-current and open-circuit-voltageconvert_2_ivsamples()already applies a simple harvesting-algo and creates ivsamples./examples/example_convert_ivonne.pyconverts IVonne recording (jogging_10m.iv) to shepherd ivcurves, NOTE: slow implementationexample_extract_logs.pyis analyzing all files in directory, saves logging-data and calculates cpu-load and data-rateexample_generate_sawtooth.pyis using Writer to generate a 60s ramp with 1h repetition and uses Reader to dump metadata of that fileexample_plot_traces.pydemos some mpl-plots with various zoom levelsexample_repair_recordings.pymakes old recordings from shepherd 1.x fit for v2jogging_10m.iv50 Hz measurement with Short-Circuit-Current and two other parametersrecorded with "IVonne"Functionality Update (WIP)Core.Reader__repr__()read_buffersget_calibration_dataget_window_samplesget_modeget_configget_hostnameget_datatypeget_hrv_configis_validenergy()check_timediffs()get_metadata()save_metadata()Writer(Reader)(core, data are the same)append_iv_data_rawappend_iv_data_sistore_configstore_hostnamedata.Reader(CoreReader)save_csv()save_log()downsample()resample()generate_plot_data()assemble_plot()plot_to_file()multiplot_to_file()CLI-InterfaceAfter installing the module the datalib offers some often needed functionality on the command line:Validate Recordingstakes a file or directory as an argumentshepherd-datavalidatedir_or_file# examples:shepherd-datavalidate./
shepherd-datavalidatehrv_saw_1h.h5Extract IV-Samples to csvtakes a file or directory as an argumentcan take down-sample-factor as an argumentshepherd-dataextract[-fds-factor][-sseparator_symbol]dir_or_file# examples:shepherd-dataextract./
shepherd-dataextract-f1000-s;hrv_saw_1h.h5Extract meta-data and sys-logstakes a file or directory as an argumentshepherd-dataextract-metadir_or_file# examples:shepherd-dataextract-meta./
shepherd-dataextract-metahrv_saw_1h.h5Plot IVSamplestakes a file or directory as an argumentcan take start- and end-time as an argumentcan take image-width and -height as an argumentshepherd-dataplot[-sstart_time][-eend_time][-wplot_width][-hplot_height][--multiplot]dir_or_file# examples:shepherd-dataplot--multiplot./
shepherd-dataplot-s10-e20hrv_saw_1h.h5Downsample IVSamples (for later GUI-usage, TODO)generates a set of downsamplings (20 kHz to 0.1 Hz in x4 to x5 Steps)takes a file or directory as an argumentcan take down-sample-factor as an argumentshepherd-datadownsample[-fds-factor][-rsample-rate]dir_or_file# examples:shepherd-datadownsample./
shepherd-datadownsample-f1000hrv_saw_1h.h5
shepherd-datadownsample-r100hrv_saw_1h.h5Data-Layout and Design choicesDetails about the file-structure can be found in themain-project.TODO:update design of filedata dtype, mode, ...Modes and DatatypesModeharvesterrecorded a harvesting-source like solar with one of various algorithmsDatatypeivsampleis directly usable by shepherd, input for virtual source / converterDatatypeivcurveis directly usable by shepherd, input for a virtual harvester (output are ivsamples)Datatypeisc_vocis specially for solar-cells and needs to be (at least) transformed into ivcurves laterModeemulatorreplayed a harvester-recording through a virtual converter and supplied a target while recording the power-consumptionDatatypeivsampleis the only output of this modeCompression & Beaglebonesupported are uncompressed, lzf and gzip with level 1 (order of recommendation)lzf seems better-suited due to lower load, or if space isn't a constraint: uncompressed (None as argument)note: lzf seems to cause trouble with some third party hdf5-toolscompression is a heavy load for the beaglebone, but it got more performant with recent python-versionssize-experiment A: 24 h of ramping / sawtooth (data is repetitive with 1 minute ramp)gzip-1: 49'646 MiB -> 588 KiB/slzf: 106'445 MiB -> 1262 KiB/suncompressed: 131'928 MiB -> 1564 KiB/scpu-load-experiments (input is 24h sawtooth, python 3.10 with most recent libs as of 2022-04)warning: gpio-traffic and other logging-data can cause lots of loademu_120s_gz1_to_gz1.h5 	-> emulator, cpu_util [%] = 65.59, data-rate =  352.0 KiB/s
  emu_120s_gz1_to_lzf.h5 	-> emulator, cpu_util [%] = 57.37, data-rate =  686.0 KiB/s
  emu_120s_gz1_to_unc.h5 	-> emulator, cpu_util [%] = 53.63, data-rate = 1564.0 KiB/s
  emu_120s_lzf_to_gz1.h5 	-> emulator, cpu_util [%] = 63.18, data-rate =  352.0 KiB/s
  emu_120s_lzf_to_lzf.h5 	-> emulator, cpu_util [%] = 58.60, data-rate =  686.0 KiB/s
  emu_120s_lzf_to_unc.h5 	-> emulator, cpu_util [%] = 55.75, data-rate = 1564.0 KiB/s
  emu_120s_unc_to_gz1.h5 	-> emulator, cpu_util [%] = 63.84, data-rate =  351.0 KiB/s
  emu_120s_unc_to_lzf.h5 	-> emulator, cpu_util [%] = 57.28, data-rate =  686.0 KiB/s
  emu_120s_unc_to_unc.h5 	-> emulator, cpu_util [%] = 51.69, data-rate = 1564.0 KiB/s