What is de-facto File Format for Storing/Transporting Digital Signal Data? - Page 1
Forum section: electronics / projects

author: Mechatrommer
author_rank: Super Contributor
timestamp: March 09, 2016, 05:05:53 am
content: for storing/transporting/tranferring digital signal data in PC. such as DSO data, DAQ data and the like. i can think of several type:1) fixed-bits-length such as 8 bits DSO, 12 bits DAQ etc2) fixed-byte floating points databoth can do. preferably the file format standard and description is available online to be implemented in a software application (import/export). please advice if you know any, thanks.

author: coppice
author_rank: Super Contributor
timestamp: March 09, 2016, 05:17:56 am
content: The most widely used file formats for waveforms are the formats originally designed for audio, like .wav files.

author: Mechatrommer
author_rank: Super Contributor
timestamp: March 09, 2016, 06:21:32 am
content: sorry i forgot to mention, the format should contains critical information about the acquisition property of the signal such as sampling rate, scaling and offset if anything applicable... and also not the CSV ascii text based. storing hundreds of Mega Sample points in CSV will required Giga Bytes of storage and super-inefficient transcoding ascii to float numbers.Quote from: coppice on March 09, 2016, 05:17:56 ami suspect this is fix-bit-length type, not floating point type? although, i struggled a long time ago to find the format specification. can it be extended to store floating points numbers?

author: Armxnian
author_rank: Regular Contributor
timestamp: March 09, 2016, 06:47:32 am
content: .wav is a versatile format. Can store 32bit signed integer and 64bit float, everything in between and various other encodes for audio.

author: BravoV
author_rank: Super Contributor
timestamp: March 09, 2016, 06:51:12 am
content: If I'm not mistaken the max file size of wav is 4 GB ?

author: ChunkyPastaSauce
author_rank: Unknown
timestamp: March 09, 2016, 07:20:05 am
content: RF64 is extended version of .wav ... stores 16 exabytes...The file format is going to depended on # of channels, whether each channel is independent of another or not (for example, different time reference for each signal), variable vs constant rate sampling, real-timeness and jitter, read-while-write, etc...

author: coppice
author_rank: Super Contributor
timestamp: March 09, 2016, 07:28:03 am
content: Quote from: Mechatrommer on March 09, 2016, 06:21:32 amYou suspect wrong. For linear data you can store 8, 16, 24, or 32 bit signed or unsigned integer samples. Floating samples are also supported. I am not sure about double precision floating point. Various compressed formats can also be stored in wave files. You can have multiple parallel channels. The header contains information about the sample type, the number of channels, etc. Its actually quite flexible.Traditional .wav files have a 4GB limit. Some systems limit you to 2GB, because some people don't seem to understand the difference between signed and unsigned. As others have said, there is an extended .wav format for the 64 bit world. Unfortunately Its not nearly as widely implemented.

author: Karel
author_rank: Super Contributor
timestamp: March 09, 2016, 09:40:43 am
content: I'm not aware of a defacto file format for time series data for test & measurement.However, there's a defacto standard for biomedical time series data, which can be used also for general test & measurement like DSO's.The name of this format is EDF (European Data Format).The advantage of EDF is that there are lot's of tools and viewers available that can read/write those files, like Scilab, R, Octave, Matlab, Polyman, EDFbrowser, etc.There's also a free C/C++ library to read/write EDF.The pro's of EDF: it's a simple format that's easy to implement. The header contains all necessary data to store sensitivity, samplerate, channel name, etc.Also, it supports channels with different samplerate.Cons: EDF stores the samples as 16-bit 2'complements while you probably need only 8 bits resolution.Technically it doesn't make a difference, apart from the resulting filesize...https://en.wikipedia.org/wiki/European_Data_Format

author: Mechatrommer
author_rank: Super Contributor
timestamp: March 09, 2016, 10:17:09 am
content: Quote from: coppice on March 09, 2016, 07:28:03 amyeah as i've been educated that *.wav is a song file format to hear musicsQuote from: Karel on March 09, 2016, 09:40:43 ami have to pay $35.95 just to get to the standard sigh

author: jeremy
author_rank: Super Contributor
timestamp: March 09, 2016, 10:27:25 am
content: I normally use HDF5, and have recorded traces in the hundreds of gigabytes without issue. HDF5 is very common in the scientific computing and high performance computing areas.

author: HAL-42b
author_rank: Frequent Contributor
timestamp: March 09, 2016, 10:29:59 am
content: Quote from: Karel on March 09, 2016, 09:40:43 amThis looks really useful. Thanks for sharing.Extending it with different bit depths might be useful.

author: Karel
author_rank: Super Contributor
timestamp: March 09, 2016, 10:30:40 am
content: Wave format:Pros: Common format, easy to implement, supports 8-bit samples (smaller filesize).Cons: no possibility to store: sensitivity/scaling, offset, units (Volts, milli-Volts, amps, etc.), description of measurement/experiment.

author: Karel
author_rank: Super Contributor
timestamp: March 09, 2016, 10:34:28 am
content: Quote from: Mechatrommer on March 09, 2016, 10:17:09 amThe EDF format is a truly open format and available for free without any registration.Here's the link to the one and only official EDF format website:http://edfplus.info/

author: Karel
author_rank: Super Contributor
timestamp: March 09, 2016, 10:38:20 am
content: Quote from: HAL-42b on March 09, 2016, 10:29:59 amPlease don't divert from the (defacto) standards in order to maintain compatibility.There's already a 24-bit version of EDF which is called BDF:http://www.biosemi.com/faq/file_format.htmhttp://www.teuniz.net/edfbrowser/bdfplus%20format%20description.html

author: Mechatrommer
author_rank: Super Contributor
timestamp: March 09, 2016, 11:14:23 am
content: is there a need for bigger than say 4 bytes float, or 8 bytes double? for time series data? like from DSO or DMM data? what measurement can be greater than 3.402823E38 and smaller than 1.401298E-45 in magnitude for both sign? is there a format that can differentiate, or store both interleaved format and non-interleaved format, ie 1 channel in a chunk followed by another channel chunk? and then back to channel1's chunk again? wav specification and terms are more towards audio data, EDF is specific to medical data, it even has its own "patience name" field in it, and most header fields are Ascii based, there is good in this since its "endian-free" format, but does the world separated into two big endian group differences? HDF5 sound application-blind, i mean its generically explain it simply as "data model", but is downloaded in "kit", "api" and "tools". why do i have to download all that to implement a storage format? does EDF, WAV and HDF5 has "time-frame" field so we can synchronize data from another channel or even from another files?

author: Karel
author_rank: Super Contributor
timestamp: March 09, 2016, 11:41:19 am
content: Quote from: Mechatrommer on March 09, 2016, 11:14:23 amWhy do you think this is a problem?Focus on the availability of tools and libraries and on the possibilty to easily open it in common programs for analysis.

author: nctnico
author_rank: Super Contributor
timestamp: March 09, 2016, 12:20:42 pm
content: I'd go for CSV (or any text base format) because it is self documenting. Any binary format is prone to interpretation differences and loss of readability. The length limit of a WAV file is another typical problem you'll have with a binary format. Actually a text based format can be more size efficient because you only store relevant information (significant digits) versus enough room for any number.

author: dom0
author_rank: Super Contributor
timestamp: March 09, 2016, 12:26:49 pm
content: HDF5Widely used, good software support, supports practically all data types and dimensions of data, supports essentially arbitrary meta data.It might be a bit "heavyweight" if all you want to do is store a handful of DSO traces or something like that.

author: Karel
author_rank: Super Contributor
timestamp: March 09, 2016, 12:35:24 pm
content: Quote from: nctnico on March 09, 2016, 12:20:42 pmText based formats are much bigger (multiple times) compared to binary formats.For example: the value -1.2345 takes 7 bytes plus the CSV separator character plus the newline character is 9 bytes in total.The same value can be stored in a binary format using only 2 bytes.

author: jeremy
author_rank: Super Contributor
timestamp: March 09, 2016, 01:08:58 pm
content: I don't see anything mentioning what language you are using, but for Python I use h5py. It's very simple to use compared to the C library and treats the hdf5 file like a collection of numpy arrays; the exact numerical format is up to you. It also automatically chunks files, checksums and compresses them if you enable the right options.Without compression but with checksums, I was able to easily stream data to an ssd at 350MB/s while remaining inside python in less than 100 lines of code

author: Mechatrommer
author_rank: Super Contributor
timestamp: March 09, 2016, 05:13:24 pm
content: Quote from: nctnico on March 09, 2016, 12:20:42 pmits self documenting until you try to save or read starting 24 Mega Points and up... by that time its not self documenting but self annoying. you havent saved 1Mpts of CSV from rigol yetQuote from: jeremy on March 09, 2016, 01:08:58 pmany language the one can name. the format should be language-independent shouldnt it?

author: dom0
author_rank: Super Contributor
timestamp: March 09, 2016, 05:14:29 pm
content: "self documenting" and "cat <file> is self explanatory" are two different things.Although I concur that CSV isn't really well suited for larger amounts of data.

author: Mechatrommer
author_rank: Super Contributor
timestamp: March 09, 2016, 05:26:17 pm
content: Quote from: jeremy on March 09, 2016, 01:08:58 pmcome to think of it, its now sensical. it must be one complicated format, that need somehow somesort of adaptive algorithm to compress, stream etc the data, probably tuning to the best machine and multithreading config its running in. the drawback is we are tied with the system, i'll check back again when i have time, thanks for the suggestion...