Package: pyjls
JLSWelcome to theJoulescope®File Format project.
The goal of this project is to provide performant data storage for huge,
simultaneous, one-dimensional signals. This repository contains:The JLS file format specificationThe implementation in CLanguage bindings for PythonFeaturesCross-platformMicrosoft Windows x64Apple macOS x64Apple macOS ARMLinux x64Linux aarch64 (ARM 64-bit).  Supports Raspberry Pi 4.Support for multiple, simultaneous data sourcesSupport for multiple, simultaneous signal waveformsFixed sample rate signals (FSR)Handles missing samples gracefully (interpolate) 🔜Multiple data types including:Floating point: f32, f64Unsigned integers: u1, u4, u8, u16, u24, u32, u64Signed integers: i4, i8, i16, i24, i32, i64Fixed-point, signed integers (same bit sizes as signed integers)Boolean (digital) 1-bit signals = u1Variable sample rate (VSR) signals 🔜Fast read performanceSignal Summaries"Zoomed out" view with mean, min, max, standard deviationProvides fast waveform load without any additional processing stepsAutomatic load by summary levelFast seek, next, previous accessSample ID to Wall-clock time (UTC) for FSR signalsAnnotationsGlobal VSR annotationsSignal annotations, timestamped to sample_id for FSR and UTC time for VSRSupport for text, marker, and user-defined (text, binary, JSON)User dataArbitrary data included in the same fileSupport for text, binary, and JSONReliabilityIntegrated integrity checks using CRC32CFile data still accessible in the case of improper program termination 🔜Uncorrupted data is still accessible in presence of file corruption 🔜Write once, except for indices and the doubly-linked list pointersCompression options 🔜lossless 🔜lossy 🔜lossy with downsampling below threshold 🔜Support level 0 DATA not written (only INDEX & SUMMARY) 🔜Items marked with 🔜 are under development and coming soon.
Items marked with ⏳ are planned for future release.As of June 2023, the JLS v2 file structure is well-defined and stable.
However, the compression storage formats are not yet defined and
corrupted file recovery is not yet implemented.Why JLS?The world is already full of file formats, and we would rather not create
another one.  However, we could not identify a solution that met these
requirements.HDF5meets the
large storage requirements, but not the reliability and rapid load requirements.
TheSaleae binary export file format v2is also not suitable since it buffers stores single, contiguous blocks.Sigrok v2is similar.
TheSigrok v3format
(under development as of June 2023) is better in that it stores sequences of
"packets" containing data blocks, but it still will does not allow for
fast seek or summaries.Timeseries databases, such asInfluxDB, are
powerful tools.  However, they are not well-designed for fast sample-rate
data.Media containers are another option, especially the ISO base media file format
used by MPEG4 and many others:ISO/IEC 14496-14:2020 SpecificationOverviewHowever, the standard does not include the ability to store the signal summaries
and our specific signal types.  While we could add these features, these formats
are already complicated, greatly reducing the advantage of repurposing them.Why JLS v2?This file format is based upon JLS v1 designed forpyjoulescopeand used by theJoulescopetest instrument.  We leveraged
the lessons learned from v1 to make v2 better, faster, and more extensible.The JLS v1 format has been great for the Joulescope ecosystem and has
accomplished the objective of long data captures (days) with fast
sampling rates (MHz).  However, it now has a long list of issues including:Inflexible storage format (always current, voltage, power, current range, GPI0, GPI1).Unable to store from multiple sources.Unable to store other sources and signals.No annotation support:41,93.Inflexible user data support.Inconsistent performance across sampling rates, zoom levels, and file sizes:48,103.Unable to correlate sample times with UTC:55.The JLS v2 file format addressed all of these issues, dramatically
improved performance, and added new capabilities, such as signal compression.How?At its lowest layer, JLS is an enhancedtag-length-value(TLV)
format. TLV files form the foundation of many reliable image and video formats,
including MPEG4 and PNG.  The enhanced header contains additional fields
to speed navigation and improve reliability.  The JLS file format calls
each TLV achunk.  The enhanced tag-length component thechunk headeror simplyheader.  The file also contains afile header, not to be
confused with thechunk header.  Achunkmay have zero payload length,
in which case the next header follows immediately.  Otherwise, achunkconsists of aheaderfollowed by apayload.The JLS file format definessourcesthat produce data.  The file allows
the application to clearly define and label the source.  Each source
can have any number of associated signals.Signalsare 1-D sequences of values over time consisting of a single,
fixed data type.  Each signal can have multipletracksthat contain
data associated with that signal. The JLS file supports two signal types:
fixed sample rate (FSR) and variable sample rate (VSR).  FSR signals
store their sample data in the FSR track using FSR_DATA and FSR_SUMMARY.
FSR time is denoted by samples using timestamp.  FSR signals also support:Sample time to UTC time mapping using the UTC track.Annotations with the ANNOTATION track.VSR signals store their sample data in the VSR track.  VSR signals
specify time in UTC (wall-clock time).  VSR signals also
support annotations with the ANNOTATION track.
The JLS file format supports VSR signals that only use the
ANNOTATION track and not the VSR track.  Such signals are commonly
used to store UART text data where each line contains a UTC timestamp.Signals support DATA chunks and SUMMARY chunks.
The DATA chunks store the actual sample data.  The SUMMARY chunks
store the reduced statistics, where each statistic entry represents
multiple samples.  FSR tracks store the mean, min, max,
and standard deviation.  Although standard deviation requires the
writer to compute the square root, standard deviation keeps the
same units and bit depth requirements as the other fields.  Variance
requires twice the bit size for integer types since it is squared.Before each SUMMARY chunk, the JLS file will contain the INDEX chunk
which contains the starting time and offset for each chunk that
contributed to the summary.  This SUMMARY chunk enables fast O(log n)
navigation of the file.  For FSR tracks, the starting time is
calculated rather than stored for each entry.The JLS file format design supports SUMMARY of SUMMARY.  It supports
the DATA and up to 15 layers of SUMMARIES.  timestamp is given as a
64-bit integer, which allows each summary to include only 20 samples
and still support the full 64-bit integer timestamp space.  In practice, the
first level summary increases a single value to 4 values, so summary
steps are usually 50 or more.Many applications, including the Joulescope UI, prioritize read performance,
especially visualizing the waveform quickly following open,
over write performance.   Waiting to scan through a 1 TB file is not a
valid option.  The reader opens the file and scans for sources and signals.
The application can then quickly load the highest summary of summaries
for every signal of interest.  The application can very quickly display this
data, and then start to retrieve more detailed information as requested.Example file structuresof
header
USER_DATA(0, NULL)    // Required, point to first real user_data chunk
SOURCE_DEF(0)         // Required, internal, reserved for global annotations
SIGNAL_DEF(0, 0.VSR)  // Required, internal, reserved for global annotations
TRACK_DEF(0.VSR)
TRACK_HEAD(0.VSR)
TRACK_DEF(0.ANNO)
TRACK_HEAD(0.ANNO)
SOURCE_DEF(1)         // input device 1
SIGNAL_DEF(1, 1, FSR) // our signal, like "current" or "voltage"
TRACK_DEF(1.FSR)
TRACK_HEAD(1.FSR)
TRACK_DEF(1.ANNO)
TRACK_HEAD(1.ANNO)
TRACK_DEF(1.UTC)
TRACK_HEAD(1.UTC)
USER_DATA           // just because
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_DATA(1.FSR)
TRACK_INDEX(1.FSR, lvl=0)
TRACK_SUMMARY(1.FSR, lvl=1)
TRACK_INDEX(1.FSR, lvl=1)
TRACK_SUMMARY(1.FSR, lvl=2)
USER_DATA           // just because
END
eofNote that TRACK_HEAD(1.FSR) points to the first TRACK_INDEX(1.FSR, lvl=0) and
TRACK_INDEX(1.FSR, lvl=1).
Each TRACK_DATA(1.FSR) is in a doubly-linked list with its next and previous
neighbors.  Each TRACK_INDEX(1.FSR, lvl=0) is likewise in a separate doubly-linked
list, and the payload of each TRACK_INDEX points to the summarized TRACK_DATA
instances.  TRACK_INDEX(1.FSR, lvl=1) points to each TRACK_INDEX(1.FSR, lvl=0) instance.
As more data is added, the TRACK_INDEX(1.FSR, lvl=1) will also get added to
the INDEX chunks at the same level.Resourcessource codedocumentationpypiJoulescope(Joulescope web store)forumReferencesJLS v1:lower-layer,upper-layer.Sigrok/v3, which shares
many of the same motivations.Tag-length-value:Wikipedia.Doubly linked list:Wikipedia.LicenseThis project is Copyright © 2017-2023 Jetperch LLC and licensed under the
permissiveApache 2.0 License.