Software to interactively plot a large number of X-Y points (>100k) - Page 1
Forum section: electronics / general

author: jeremy
author_rank: Super Contributor
timestamp: February 27, 2019, 09:22:01 am
content: Hi all,I've got some logged data (temperature, humidity, resistance, etc) which I'd like to be able to view and zoom in/out of in an interactive manner. Number of points is somewhere between 100k and 500k. Does anyone know of any software that lets you view this sort of data (with appropriate downsampling), and also lets you zoom in on any interesting bits? I really like the PicoLog software, but it only works with their proprietary sensors, not random data from disk.I can reprocess the data into CSV or some other format if necessary.

author: RoGeorge
author_rank: Super Contributor
timestamp: February 27, 2019, 09:29:50 am
content: Would SPICE (e.g. LTSpice, QUCS) charting capabilities be enough?If yes, you can define a CSV voltage source, then chart it.Later edit:This one was doing nice charts, too, but didn't tried the latest versions.https://hackaday.io/project/5334-serialplot-realtime-plotting-software

author: eliocor
author_rank: Unknown
timestamp: February 27, 2019, 12:51:07 pm
content: DPLOT:https://www.dplot.com/index.htmultrafast even with LOTS of data!

author: Selectech
author_rank: Regular Contributor
timestamp: February 27, 2019, 01:44:34 pm
content: KST.https://kst-plot.kde.org/I use it for large data sets from sensors collected by embedded MCU applications.

author: ramon
author_rank: Regular Contributor
timestamp: February 27, 2019, 01:46:55 pm
content: RRDtool + Javascript RRDGo here ->http://javascriptrrd.sourceforge.net/Do not just look the pictures ... click on the images.Click + move the mouse to do zoom into the graphs. (I guess that this is maybe what you want).This tools have been used for years for network traffic and server/cpu stats (usually), but actually they can 'eat' any other kind of data series too. They require a big learning curve.

author: Karel
author_rank: Super Contributor
timestamp: February 27, 2019, 03:18:40 pm
content: EDFbrowser can import CSV/ASCII.https://www.teuniz.net/edfbrowser/

author: HighVoltage
author_rank: Super Contributor
timestamp: February 27, 2019, 04:04:38 pm
content: For statistical analysis of large data sets I use MINITABhttp://www.minitab.com/en-us/

author: MarkL
author_rank: Unknown
timestamp: February 27, 2019, 04:31:46 pm
content: I do all my plotting and statistical crunching in Octave. It's the open equivalent of Matlab:http://octave.orgA quick test of a scatter plot of 500k random (x,y) points takes about 3 seconds on a 2.5GHz processor. You can then pan and zoom in/out. I don't know if that's fast enough for you compared to some of the other packages mentioned, but I don't usually have that many data points to worry about.It has CSV and other import formats.

author: rhb
author_rank: Super Contributor
timestamp: March 01, 2019, 11:59:28 pm
content: I use gnuplot for such things. No connection to Stallman's Gnu project. I've done a very large fraction of my data analysis work with it. The Marquardt-Levenberg solver in it is the best I've ever used. It does need a fast graphics card for manipulating large cloud points.My experience with large datasets (i.e. TB) is that color XY plots and XYZ movies are the most useful. Points get obscured by other points if you do clouds in XYZ. In particular probability density functions (PDFs) are very valuable. Most large databases are full of garbage. Theyreally stand out in a PDF of the dataset.There is a bunch of code I wrote for this in the Seismic Unix repository on Github that John Stockwell has set up. I've also got a code I've never published which computes spatial statistical attributes and filtering on TB size data sets. Sorry, I can't tell you the names of the programs in Seismic Unix. Stockwell changed the names and stripped my copyright from the codes. So I quit doing support after 10-15 years.

author: fki82
author_rank: Contributor
timestamp: March 05, 2019, 06:42:03 pm
content: Can you pan and zoom with any of the mentioned tools interactively?

author: rhb
author_rank: Super Contributor
timestamp: March 05, 2019, 06:47:41 pm
content: Yes, gnuplot provides pan and zoom. SU may just offer zoom. I can't recall as it's been a few years since I used it last.

author: Karel
author_rank: Super Contributor
timestamp: March 06, 2019, 07:33:21 am
content: Quote from: fki82 on March 05, 2019, 06:42:03 pmYes, EDFbrowser can pan, zoom in/out, has filters and FFT.

author: BFX
author_rank: Frequent Contributor
timestamp: March 06, 2019, 03:18:02 pm
content: Grafana is the best for your purpose.https://grafana.com

author: KaneTW
author_rank: Frequent Contributor
timestamp: March 07, 2019, 04:25:41 pm
content: I use Grafana for presenting interactive time-series dataFor offline data, my usual choice of CAS (Mathematica) does about 1mn points in 3.5sec.

author: pwlps
author_rank: Frequent Contributor
timestamp: March 15, 2019, 07:07:39 am
content: I'm using Igor Pro for all my data analysis. The software is designed to manipulate very large data sets and is really fast. On my i5-3470 plotting 10M points takes less than 1 second. You can pan and zoom using the mouse. You can control all basic functions either with menus or from the command line. What I like is that when using the menus the corresponding commands are generated and go into the history record so it is easy to recall them from there, you can also grab them from the history window and copy to a macro if you want to automate a sequence without learning the details of the programming language. Igor Pro forum is very active and it's easy to get help there.