mplot: logging and plotting script for long term measurements - Page 1
Forum section: electronics / general

author: Muxr
author_rank: Super Contributor
timestamp: May 29, 2017, 06:49:46 pm
content: I shared a plotting and logging script for long term measurements with the folks over on the USA calibration club thread:https://www.eevblog.com/forum/metrology/usa-calibration-club/msg1219008/#msg1219008Currently it's been tested on Keithley 2015, but should likely work with other meters from the Keithley family, via serial rs232 interface. I will probably add more support over time, but any instrument should work with minor tweaks possibly.Most likely only this function should need modification:https://github.com/sirmo/muxrplot/blob/master/mplot/logger.py#L94Here is an example of it in action:The script was written using Python and python libraries like, scipy, numpy, seaborn and pyserial. It's only tested to work on Unix-y systems, but it should be possible to make it work under Windows.If you're on Windows probably the easiest is to get a VirtualBox VM of Linux going, VirtualBox allows passing of a USB interface to the guest for instance. The other option, which is the main reason why I decided to write this for Unix is that it should work on Raspberry Pi. I would like to have the ability for instance to connect multiple remote instruments to a R Pi and have it log all the measurements, I can just then login remotely over wifi if I need to fetch the data.Anyways, the script is hopefully fairly simple and should be easy to modify to your needs.The project can be found here:https://github.com/sirmo/muxrplottogether with the install instructions.I will be monitoring this thread, so if you need assistance with modifications to make it work for your needs I am willing to help.

author: vindoline
author_rank: Unknown
timestamp: May 29, 2017, 09:24:03 pm
content: Since this is the "metrology" section, we should probably fix the Y-axis labels so the don't imply pico-volt sensitivity! (Unless you do have an 11.5 digit meter)

author: Muxr
author_rank: Super Contributor
timestamp: May 29, 2017, 09:42:11 pm
content: Quote from: vindoline on May 29, 2017, 09:24:03 pmThat's a good point, will work on that one! Cheers.

author: Muxr
author_rank: Super Contributor
timestamp: May 29, 2017, 10:56:42 pm
content: It's been fixed:It defaults to 7 least significant digits. I also added an option so you can tweak it when running graphs.Code:> mplot -hmplotusage: mplot [-h] [-t TITLE] [-y YDIGITS]  positional arguments:infileoutfileoptional arguments:-h, --help show this help message and exit-t TITLE, --title TITLEtitle to be used in the chart-y YDIGITS, --ydigits YDIGITSNumber of least significant digits in the Y labels

author: cellularmitosis
author_rank: Unknown
timestamp: May 29, 2017, 11:32:47 pm
content: This is great! I'll definitely be using this myself!

author: cellularmitosis
author_rank: Unknown
timestamp: June 03, 2017, 08:46:37 pm
content: Just got a Si7021 breakout board working last night. I used your script to plot the temperature and humidity.The temperature is purple at the bottom of the graph (in C), and the humidity is in cyan at the top of the graph (in %).Interesting to see how variable the humidity is -- I'm pretty sure you're seeing the frequency at which my A/C came on. I think the low dip at the beginning was when I adjusted the thermostat lower before going to bed (it ran longer and lowered the humidity further), and the drift at the end is maybe when I bumped up the thermostat a couple of degrees, so the humidity was no longer being controlled.

author: cellularmitosis
author_rank: Unknown
timestamp: June 03, 2017, 08:50:19 pm
content: (might as well post the code I used to log the data. I had some fun using a binary protocol with CRC between the arduino and PC.)

author: Muxr
author_rank: Super Contributor
timestamp: June 03, 2017, 10:55:18 pm
content: Nice!Yup, one of the main reasons builders don't recommend oversized AC units is to control the humidity better, when they are on for longer periods of time the humidity drops. I actually have an additional dehumidifier in my house and my humidity is usually around 40%, outside humidity is like 80%-90% (Florida).High humidity does mean you pretty much don't have to worry about ESD though

author: cellularmitosis
author_rank: Unknown
timestamp: July 14, 2017, 11:01:34 pm
content: You got your first pull request bubI ran into an error with matplotlib when I tried to plot several days worth of data from an LTC2400 (over 1 million data points):"OverflowError: Allocated too many blocks"It turns out you can configure matplotlib to consider the data in chunks when it performs the plot, which avoids this error for large data sets (seehttps://stackoverflow.com/questions/20330475/matplotlib-overflowerror-allocated-too-many-blocks/23361090#23361090).

author: Muxr
author_rank: Super Contributor
timestamp: July 15, 2017, 02:15:44 am
content: Nice find @cellularmitosis! Thank you! Merged.

author: cellularmitosis
author_rank: Unknown
timestamp: July 15, 2017, 02:24:15 am
content: Here's a PR to add a rolling-average filter:https://github.com/sirmo/muxrplot/pull/2

author: Muxr
author_rank: Super Contributor
timestamp: July 15, 2017, 02:36:36 am
content: Quote from: cellularmitosis on July 15, 2017, 02:24:15 amSweet! Merged!

author: Muxr
author_rank: Super Contributor
timestamp: July 17, 2017, 03:58:57 am
content: In case anyone has a Fluke 8845/8846 I wrote a quick python class to interface with it:https://gist.github.com/sirmo/8096cc1fef598762494e3d987befbfefIt will open a connection to the given ip/port, set NPLC 60 and blank out the screen (to save from VFD burnout).

author: alm
author_rank: Super Contributor
timestamp: July 17, 2017, 05:31:40 am
content: Have you considered using an existing library likepython-ivifor the instrument drivers instead of rolling your own? Or is that interface insufficient for the features you need?

author: Muxr
author_rank: Super Contributor
timestamp: July 17, 2017, 05:16:09 pm
content: I suppose I could give it a try (totally forgot about it). They don't seem to list Fluke 884* models on the supported list though.

author: alm
author_rank: Super Contributor
timestamp: July 17, 2017, 11:11:22 pm
content: As far as far as I know they do not support the Fluke. But implementing an existing interface may be more productive than rolling your own by better facilitating reuse.

author: Muxr
author_rank: Super Contributor
timestamp: July 18, 2017, 12:10:51 am
content: I can look at writing a PR for it, sure. I just wrote that snippet in 10 minutes really. A Fluke 8846 driver for python-ivi will take much longer than that.

author: alm
author_rank: Super Contributor
timestamp: July 18, 2017, 12:47:58 am
content: True, although an IVI driver for basic functionality should not take that much code (much of it is handled by the scpi.dmm classes). I was mostly thinking the other way around: if you want your logger to interface with other meters (e.g. GPIB, TCP/IP or USB), then using an existing back-end like python-ivi, or sigrok (seems mostly aimed at handhelds) or some other abstraction makes sense to me, rather than inventing your own. Granted, DMM support in python-ivi is currently limited to a few Agilent meters and one Rigol (in Agilent emulation mode), so the short-term payoff is quite limited.Python-ivi is the direction I plan to go if I ever get around to rewriting my logging software. Currently it is using a heavily-modified version ofgpib-devicesbut that is not actively maintained and heavily tied to GPIB (instruments inherit from Device488). Since IVI is an industry standard, it seems like a logical choice. The other abstractions I looked at were more limited or seemed aimed at different types of equipment.

author: Muxr
author_rank: Super Contributor
timestamp: July 18, 2017, 01:30:43 am
content: @alm, that's a good idea. I will look into this further. I would like to abstract the driver part of it away for sure, will play with ivi over the weekend, and see if I can add Fluke8846 and hp3458a to it (my Keithley 2015 bit the dust).The other idea I've been sort of toying with as far as the metrics side is concerned, is using something like TSDB to store all the metrics and maybe use a front end like Grafana. Like everyone else I suppose I suffer from too many projects so it's hard to commit to a bigger idea. This was just a quick effort to get the plotting going for our little USA calibration club.

author: cellularmitosis
author_rank: Unknown
timestamp: July 18, 2017, 01:35:18 am
content: Quote from: Muxr on July 18, 2017, 01:30:43 amOoooooh, interesting.http://opentsdb.net/Edit: Holy cow!https://misfra.me/2016/04/09/tsdb-list/

author: alm
author_rank: Super Contributor
timestamp: July 18, 2017, 01:51:51 am
content: Getting the 8846 to work should be straight-forward, since its SCPI interface will be quite similar to the Agilent 344xx meters. The 3458A might be more work since it does not implement SCPI, so you would inherit from the base dmm classes instead. The agilent.agilent437B driver might be vaguely similar in that it implements a pre-SCPI protocol (it is a power meter, rather than a DMM, though). The scpi.dmm classes might be another source.TSDB definitely looks cool if you are talking long-term measurements. My needs so far have been served by CSV files that I import in R for analysis/statistics/plotting. Not sure about a standard front-end like Grafana: they look designed for monitoring for large changes like power outages or spikes in data traffic, rather than looking for 0.000001 ppm changes through statistical analysis and relating them to atmospheric conditions.

author: Muxr
author_rank: Super Contributor
timestamp: July 18, 2017, 02:07:24 am
content: Quote from: cellularmitosis on July 18, 2017, 01:35:18 amYeah, one of the lighter ones would probably be a better idea. TSDB is awesome but it's way overkill running something like a Hadoop backend for this, unless you're doing high frequency measurements on like 100+ sources (hbase might be lighter, although haven't tried it). We use TSDB/hadoop at work but we deal with petabytes of data.Quote from: alm on July 18, 2017, 01:51:51 amGrafana supports plugins, like they even have a Plotly extension:https://grafana.com/plugins/natel-plotly-panelbut I hear you on ours being a special usecase.