Rigol ds1000z WFM file reverse engineering - Page 1
Forum section: electronics / testgear

author: Neuromodulator
author_rank: Regular Contributor
timestamp: July 18, 2016, 10:17:07 pm
content: Hello,I've been working a bit in reverse engineering the WFM file format so as to convert it to the more friendly CSV. So far the code I have looks like junk so I have no usable utility yet, but I've identified some key data fields in the file, like the time base, voltage range, active channels, and the time series. As soon as I write a program to do the WFM to CSV conversion I'll post it here. If there is interest I could post what I've found so far as well.

author: bitseeker
author_rank: Super Contributor
timestamp: July 18, 2016, 10:18:55 pm
content: Sounds like a very handy utility, Neuro. Thanks for digging into it.

author: BravoV
author_rank: Super Contributor
timestamp: July 19, 2016, 10:23:32 am
content: Neuromodulator, thank you !Subscribed.Quote from: Neuromodulator on July 18, 2016, 10:17:07 pmIf its ok with you, please post the fields details you've discovered so far, tia.

author: Neuromodulator
author_rank: Regular Contributor
timestamp: July 19, 2016, 03:52:30 pm
content: Sure, here is what I've found so far (which I haven't double checked so may contain errors):header =\(('az', 8, 'Model'),('az', 28, 'Firmware'),('i32', 88, 'Active Channels'), # bit n = Ch(n-1)('i32', 116, 'Points'),('f32', 120, 'Time Base (s)'),('ch', 124, 'Ch1'),('ch', 152, 'Ch2'),('ch', 180, 'Ch3'),('ch', 208, 'Ch4'),)channel =\(('u8', 0, 'Active'), # 0 = Yes, 0 = No('u8', 1, 'Coupling'), # 0 = DC, 1 = DC('u8', 2, 'Bandwidth Limit'), # 0 = Enabled, 1 = Disabled('i32', 4, 'Unknown'), # Changes depending on if the channel is active and some other factor('f32', 8, 'Voltage (v)'),('f32', 12, 'Position (v)'),('u8', 16, 'Inverted'), # 0 = No, 1 = Yes('u8', 17, 'Unit'), # 1 = A, 2 = V)az = ascii zeroThe channel time series begins at position 3198, and is interleaved.I'm also wondering if there is any tool to describe a binary file as a context free grammar and parse it automagically.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 19, 2016, 04:09:36 pm
content: Quote from: Neuromodulator on July 19, 2016, 03:52:30 pmautomagically? in our dream. not automagically? thats called hex editor. btw, maybe you'll find my log file helpfull (attached)

author: Neuromodulator
author_rank: Regular Contributor
timestamp: July 19, 2016, 04:43:28 pm
content: Quote from: Mechatrommer on July 19, 2016, 04:09:36 pmThanks for the file, gonna take a look at it. When I talked about parsing, I ment parsing after the grammar has been defined, not detecting the grammar. There are tools that do that, but they are tailored to computer language parsing.

author: Mechatrommer
author_rank: Super Contributor
timestamp: July 19, 2016, 05:01:27 pm
content: Quote from: Neuromodulator on July 19, 2016, 04:43:28 pmwhat grammar? its just content... binary file format. if you already know the grammar (i mean file chunk format) then you should be able to make a simple parser script. this is what i did when reverse enginerring the WFM on many files output to make content comparison.

author: Neuromodulator
author_rank: Regular Contributor
timestamp: July 19, 2016, 05:37:50 pm
content: Quote from: Mechatrommer on July 19, 2016, 05:01:27 pmI know, I was just looking for a standard way to describe a binary file grammar and a tool that could read the file and parse the binary. Since that apparently doesn't exist I'll continue doing it manually with code.

author: janoc
author_rank: Super Contributor
timestamp: July 19, 2016, 09:14:24 pm
content: Quote from: Neuromodulator on July 19, 2016, 05:37:50 pmUff, I think you are overthinking it a bit - the file format is not likely to change, so a grammar and a generic parser would be an enormous overkill for this.

author: freman
author_rank: Contributor
timestamp: December 21, 2016, 02:04:11 pm
content: I've started doing this too, how far have we all gotten?

author: TheoB
author_rank: Regular Contributor
timestamp: December 21, 2016, 09:43:31 pm
content: Just curious. Why would you want to use this method? I just ask the scope for the raw sample data. You only need a USB or network connection. It's all well documented.

author: freman
author_rank: Contributor
timestamp: December 21, 2016, 11:33:57 pm
content: It's not always convenient or possible to be hooked up to a network or computer at time of sampling

author: HardWareMan
author_rank: Newbie
timestamp: February 10, 2019, 02:06:53 pm
content: Hi! Is there any update with new WFM file?

author: preamp.org
author_rank: Contributor
timestamp: April 21, 2024, 01:36:06 pm
content: I'll do the Tread DiggerTMand revive this old thread.I have recently acquired a used DS1104Z because I needed a 4-channel scope. And of course I immediately fell down the rabbit hole of getting the captured waveform data from the scope over to the PC for post-processing. As is widely known, saving CSV data to USB-drive takes ages and thus renders the scope unusable during that time. Saving to WFM is much quicker, but that file seems to be pretty much useless, unless a close approximation is sufficient.I have not used any of the software that Rigol supplies for their scope, and I probably won't. Windows-only and ~500MB download is just not my cup of tea. But here's a question to anybody whodiduse that software: Is it actually possible to accurately convert a WFM file into a CSV file (or anything else, really) with Rigol tools, or won't even their own tools work on that files?The reason I'm asking is that I have a strong suspicion that there's some vital information missing from the files altogether. From what I have figured out so far, the RAW values in the WFM file seem to be the exact RAW values that the ADC spits out. You can clearly see the range of that data change with the vertical scale setting and the applied input signal held constant, so it is not normalized to some calibrated value. To do this, you'd need to know the calibration data of the ADC, i.e. its internal range and divider settings, and apply this manually to the RAW data to obtain the correct voltage scale. Unfortunately, that calibration data seems to be missing from the WFM file!A simple test was to make two almost-identical measurements, with only the voltage scale being a little different and thus leading to a different RAW data range. I have used 1V/Div for a resulting RAW value range of 29 to 141, and 0.9V/Div for a much bigger range of 32 to 211. The input signal was the same for both measurements, so you'd need to apply different conversion factors for both files to obtain the same resulting output voltage, obviously.Next, I have compared the WFM files with a HEX editor to look for some differing data. Apart from the obvious things like the actual decimal scale factor voltage, instrument run time and a handful of bytes that must be some kind of CRC or checksum (as those bytes change for each individual WFM file, even if you immediately save the exact same data three times in a row, without even leaving the Storage menu or doing anything else, really), there are no other differences in the file! Nothing that would relate to any kind of selected mode, or calibration data. And there certainly is no huge table that would contain all the calibration values at once. The scale factor that needs to be applied to the RAW data seems to range from 1.00 to about 1.54, and that factor is not only different for each of the four channels, but also different for each setting of the vertical scale. And not just the coarse 1-2-5-settings, but for each and every vernier step: 0.9V needs a different factor than 0.91V, or 0.92V, or 0.93V... you get the idea. Such a table would be huge. Unless there was a linear relationship between those ranges, but I have'nt taken enough measurements to try and find one.Honestly, at the moment this looks like a hopeless case. Short of manually determining the conversion factors for each channel/range-combo of interest, I don't see how this could work out. Except I'm missing something obvious... but others have tried this before, and did not succeed, either, AFAIK.

author: Mechatrommer
author_rank: Super Contributor
timestamp: April 21, 2024, 04:22:09 pm
content: Quote from: preamp.org on April 21, 2024, 01:36:06 pmVisaDSO did it, rev.eng of WFM format is far from 100% but good enough to extract some data out of it, attached is my VB translated code to decode WFM, if you still interested in this old (i would say obsolete or nonsensical) format... i would just connect my ds1000z to usb cable, much less work to do than switching usb stick back and forth... ymmv...

author: preamp.org
author_rank: Contributor
timestamp: April 21, 2024, 05:32:56 pm
content: Well, VisaDSO refused my files as unsupported.I'll have a look at your code though, thanks for posting it here.And how would you control the scope with a USB cable, using what software? That's not how I'd intend to use it, because I don't have a PC at the workspace where the scope is set up. I am more interested in using the roll mode to capture several successive minutes of data, with no real-time-PC-access necessary. No problem transferring the data with a thumb drive, but waiting half an hour for it being written as a CSV file is a real show stopper.

author: Mechatrommer
author_rank: Super Contributor
timestamp: April 21, 2024, 06:57:32 pm
content: Quote from: preamp.org on April 21, 2024, 05:32:56 pmif you send me your wfm file, i may have a look when i have time to see whats unsupported. but its been a while since i look at the code, so no promise since i can forget things.Quote from: preamp.org on April 21, 2024, 05:32:56 pmVisaDSO is not meant to control your dso, its more on downloading data to PC/laptop. but it provides limited but related control such as setting memory capture to either screen or memory. ymmv.

author: preamp.org
author_rank: Contributor
timestamp: April 22, 2024, 10:59:30 am
content: Turns out that VisaDSO refused the files from the first folder that I tried (and then I didn't bother to try some more). Those files were recorded with 6M pts. It does accept some others, though, which were recorded with only 3k pts. Unfortunately, the voltage range is way off.Please have a look at the attached example. Input signal is the same for all 4 Channels, about 6.2Vpp. They're all the same in the CSV file that the scope put out, just as you'd expect.With VisaDSO I get 4 different results, namely:5.15Vpp for CH1,5.85Vpp for CH2,3.95Vpp for CH3 and4.05Vpp for CH4.That's not exactly helpful, not even for relative comparisons between the channels, even if the absolute value was not spot-on.

author: Mechatrommer
author_rank: Super Contributor
timestamp: April 22, 2024, 12:08:08 pm
content: Quote from: preamp.org on April 22, 2024, 10:59:30 amsince as said WFM has not been fully hacked, scaling can be way off, full scale correction setting in WFM has not been found. hence VisaDSO provides manual scaling, please see attached images in order to get to manual translation (scaling and offset) setting dialog box. i've quickly calculated the scaling and offset needed forNewFile1.wfmto get as close as your screen capture and manually scaled CH1 (clicking green button CH1) please refer tomanual_scale.png, i've not scaled CH2-4 (offset left at 0 scale at 1) you can do it yourself if needed by clicking individually CH2-4 green button and ticking checkbox above them. i only provide CH1 as example. cheers.as for unsupportedNewFile3.wfm in doesntwork.7zi'll look into it later if its solvable...i've traced file incompatibility inNewFile3.wfmis due to trigger channel reported by the file as channel 6, since VisaDSO only expects maximum trigger channel as CH4, then it will stop reading the file and spit "unsuppported file" to avoid any further unexpected values in the file. may i ask what trigger setting and channel your scope was set at while doing the capture file NewFile3.wfm? i cant possibly tried all the dso setting when developing the WFM reader code in VisaDSO.

author: preamp.org
author_rank: Contributor
timestamp: April 22, 2024, 02:24:20 pm
content: Quote from: Mechatrommer on April 22, 2024, 12:08:08 pmTrigger was set to AC. That would explain CH6 on a 4-Channel scope. I've attached the settings file that was generated together with the PNG.

author: Mechatrommer
author_rank: Super Contributor
timestamp: April 22, 2024, 04:36:45 pm
content: Quote from: preamp.org on April 22, 2024, 02:24:20 pmi've added quick fix for ac sourced trigger, you can patch VisaDSO.exe with the attached file, and retry to open your NewFile3.wfm. consider this "unofficial nightly build release v1.1.4" just for you to fix just one thing. cheers.

author: preamp.org
author_rank: Contributor
timestamp: April 22, 2024, 06:13:18 pm
content: Well thanks for your effort. Unfortunately it won't run under WINE on my PC at home, so I can only test it out tomorrow at work on the Windows-machine.I don't know how or where you apply the manual conversion factor, but if you apply it directly to the RAW data, and only then do the other stuff, you'll end up with the correct voltage offset automagically. The factors for my sample file are 1.19 - 1.055 - 1.545 - 1.52 for CH1..4, respectively.I have converted the data into the LTspice RAW format. You can open the file with LTspice, go to Plot Settings > Visible Traces and add all 4 channels there. The waveform viewer lends itself nice for this purpose, since it can apply lots of math to the traces (like factors and offsets...), integrate, has two cursors, zoom, ...

author: preamp.org
author_rank: Contributor
timestamp: April 23, 2024, 06:59:02 pm
content: I have briefly tried Rigol's Ultra Scope software today. Might as well be that I missed something, but I was not able to load a WFM file with it. Looks like only CSV or something similar is supported. Which would reinforce my suspicion that not even Rigol has a tool to make sense of their own file formatEDIT: It is possible to re-load the WFM files on the scope, and convert them to CSV after the fact - while having to wait a little eternity everytime, though. Unfortunately, the scope will refuse to load a WFM file that was recorded with another scope; further strengthening my suspicion, that the necessary conversion factors never actually leave the scope.

author: preamp.org
author_rank: Contributor
timestamp: May 15, 2024, 12:27:02 pm
content: What a messTurns out that for every combination of Channel Number and Vertical Scale Setting you needtwoadditional parameters to obtain the correct result: ascale factorand anoffset. Fortunately both of them can be reconstructed from a single measurement, as long as you save it to both a WFM file and a CSV file. Then those parameters can be saved for later and applied to any subsequent WFM-only file conversions.After building a little tool that takes the tedium out of all this, I just noticed that the two parameters are actually different for AC-coupled measurements! So I'll have to take care of that, too.

author: RoGeorge
author_rank: Super Contributor
timestamp: May 15, 2024, 12:41:32 pm
content: You need to read the SCPI manual if you want to write your own, there are a few formats, and the data has to be downloaded in batches, can not be brought all at once. RAW mode means one byte for each sample, so you'll need to know volts/div and time/div to make sense of it, and a few more trigger related and probe type data to turn the raw bytes into physical values.If you need a ready made tool that can download the full memory of 24MSa in about 30 seconds, try DSremote.

author: preamp.org
author_rank: Contributor
timestamp: May 15, 2024, 01:53:11 pm
content: I am trying to convert the WFM files that get saved to a USB thumb drive, since I don't want to have a PC running alongside the scope. As far as I can tell right now, those files are missing some vital calibration data, though.

author: RoGeorge
author_rank: Super Contributor
timestamp: May 15, 2024, 02:48:48 pm
content: Never used local saved files, my oscilloscope is always connected to LAN, and using a PC when data capture or screen capture is needed. But if you have to save locally, there are many WFM readers already, first search result:https://pypi.org/project/RigolWFM/My guess is the waveform file format is something standard, like this:https://download.tek.com/manual/Waveform-File-Format-Manual-077022011.pdf

author: preamp.org
author_rank: Contributor
timestamp: May 15, 2024, 02:59:38 pm
content: Have a closer look at your first search result:https://rigolwfm.readthedocs.io/en/latest/1-DS1000Z-Waveforms.htmlSome quotes:QuoteQuoteQuoteLike I already said in my first post in this thread, there seems to be something wrong with those WFM files of a couple of specific models, and there doesn't seem to be a working conversion tool out there - not even from Rigol themselves.

author: preamp.org
author_rank: Contributor
timestamp: June 16, 2024, 09:01:10 am
content: This is even worse than I ever deemed possible. The data gets scaled differently depending on the sample rate, for the same signal on the same channel with all the other settings being the same. Why is that? Might be due to more noise on the highest sample rate, but I did not expect the difference in scale and offset to be higher than the noise level.

author: preamp.org
author_rank: Contributor
timestamp: January 12, 2025, 01:30:06 pm
content: Damn it, this "toy" sucks big time! I'll throw in the towel and get myself a proper tool for the job.Every time I try to use this thing again, there's just another edge case turning up that I was not aware of before. It's just hopeless. The calibration data, which is different for every single combination of selected channels and input range and offsets (that's why the self-calibration takes so much time after all - it has to collect a bunch of data), is missing from the WFM file and I am not aware of any way to access that data at all. This implies the following problem: It is possible to load a WFM file on the scope to review it and even use it to create a proper CSV file afterwards -- but: this will only be accurate if you haven't re-calibrated the scope in the meantime! This is just... bad