Wanted: software to view large amounts of data - Page 1
Forum section: electronics / projects

author: Harvs
author_rank: Super Contributor
timestamp: September 25, 2013, 07:15:36 am
content: I'm currently logging a lot of data for a project I'm working on (10-50 samples per sec from a dozen channels over days in duration.) I'm just currently saving it all in CSV direct from a uC.As with most data sets, 99% of it is pointless, so I need something simple to easily zoom in and out, pan, scale etc. Even better if I can overlay separate data sets and shift them relative to each other (e.g. checking for correlation of events.)I could write something myself, but I would rather something off the shelf that works well. I was surprised that about all I can find is the software suites that come with commercial DAQs, which appear locked to their product.Any suggestions?

author: quantumvolt
author_rank: Frequent Contributor
timestamp: September 25, 2013, 07:19:29 am
content: Minitab (simple) or SPSS (serious).

author: TerraHertz
author_rank: Super Contributor
timestamp: September 25, 2013, 07:29:33 am
content: You might try looking up bioinformatics software. Gene engineers have similar problems.And here's some links re data visualization. Not sure if relevant.-----------------------------Javascript D3 visualization engineSee \\Landisk\Archives\!_Tech_Info\!_Software_ref\!_Languages\Javascript\20130531_D3_visualization_tools20130531http://blog.vctr.me/posts/central-limit-theorem.htmlCentral Limit Theorem Visualized in D3In the source:<script src="http://d3js.org/d3.v3.min.js?clear"></script><script type="application/javascript">http://d3js.orgData-Driven DocumentsD3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. D3’s emphasis on web standards gives you the full capabilities of modern browsers without tying yourself to a proprietary framework, combining powerful visualization components and a data-driven approach to DOM manipulation.-------------------------------20130922https://news.ycombinator.com/item?id=6423245http://bajcic.com/blog/82/missing-drawing-apiMissing drawing API

author: EEVblog
author_rank: Unknown
timestamp: September 25, 2013, 07:40:28 am
content: With large data sets I've typically decimated the data first to get it down to a usable size using a couple of lines of code in VB or whatever, and then just used Excel to do the plotting. BUt that doesn't let you zoom right in of course.

author: David_AVD
author_rank: Super Contributor
timestamp: September 25, 2013, 08:11:31 am
content: Harvs, I can't help, but curious to know how the data is displayed?

author: ElectroIrradiator
author_rank: Frequent Contributor
timestamp: September 25, 2013, 08:36:48 am
content: Matematica 9.

author: Harvs
author_rank: Super Contributor
timestamp: September 25, 2013, 08:57:45 am
content: Quote from: David_AVD on September 25, 2013, 08:11:31 amWhat I'm essentially doing is comparing MPPT algorithms on a solar array. Six banks of panels, six load channels, etc. So it's just streams of voltage, current & power for each channel (and any variables I want to log out of the different algorithm on a regular basis to give insight into decision making). So I just want to just jump quickly into sections of the data that have interesting stuff (i.e. clouds.)Quote from: EEVblog on September 25, 2013, 07:40:28 amThat's pretty much what I've been doing up to now. It gets painful though after a while.Thanks for the suggestions, I'll check them out. D3 may be excellent, I don't know if it can do what I want yet, but it looks to be able to do it via a browser interface. That'd be nice for remote viewing the data through a web-server.

author: Stonent
author_rank: Super Contributor
timestamp: September 25, 2013, 09:01:28 am
content: I'm with Dave on Excel. Even if it's completely pointless for working with the data, you can at least filter it down significantly. If 99% of it is pointless then it should be easy to get it down to half the size or even more with a few clicks in Excel.Edit: Looks like you posted before I did. Oh well.

author: rollatorwieltje
author_rank: Unknown
timestamp: September 25, 2013, 09:42:43 am
content: If you don't mind writing your own software, take a look at the Zedgraph library for C#. At work we ran into the same issue and wrote some custom software to plot huge log files using Zedgraph.For serious stuff we use Matlab, but that is stupidly expensive.

author: David_AVD
author_rank: Super Contributor
timestamp: September 25, 2013, 09:49:08 am
content: Quote from: Harvs on September 25, 2013, 08:57:45 amWhat I was getting at is how are the data values represented? Graphs or ??

author: Codemonkey
author_rank: Regular Contributor
timestamp: September 25, 2013, 10:23:10 am
content: A slightly different approach would be to convert your data from csv into a wav file format (quick google search suggests its do-able!), then use LTspice to view the data. You can place a number of voltage and current sources in the schematic, then configure them to be Arbitrary Piece-wise linear voltage or current sources (specify the wav file as the source for the data points), and then use the waveform viewer to view your data.I've never done it myself but looks like it should be possible, and the waveform viewer in LTspice allows you to view large amounts of data points, and then zoom in on the important areas easily. It would also allow you do combine voltage and current traces into power traces etc.

author: johnwa
author_rank: Frequent Contributor
timestamp: September 25, 2013, 11:20:06 am
content: I was faced with this problem myself a while ago, viewing logs for my home automation system. Admittedly, the sample rate is only once a minute, but it still adds up to a fair bit of data.I looked around for some suitable software, but couldn't really find anything I liked. In the end, I discovered that I could use Gnuplot to read the CSV files directly and graph them. This program can be used in batch mode, taking a command file and spitting out a PNG image. I wrote a little web frontend in Python that selects the time and date range, and which channels to display, writes a series of commands out to Gnuplot, and serves up the resulting image. This lets me provide a realtime display, rather than having to import the data into a spreadsheet every time I want to look at it.Let me know if you want to give this a go - I can probably provide some code snippets.

author: HackedFridgeMagnet
author_rank: Super Contributor
timestamp: September 25, 2013, 11:38:15 am
content: I am guessing it would be easier to leave the mcu alone and process the data later.http://www.eonfusion.comhas a product that I am sure would suit but it probably wouldn't be worth buying for you, maybe you can get a demo.I must admit I would resort to code. Probably C# and use zedgraph. Maybe a database if you need a bit more speed.

author: Harvs
author_rank: Super Contributor
timestamp: September 25, 2013, 01:04:07 pm
content: Quote from: wilfred on September 25, 2013, 10:41:04 amSample rate is an absolute must, the main comparison between algorithms is how they handle transient conditions.Storing the data isn't an issue. Even doing a rough plot to find area's of interest isn't hard. It just wastes a bunch of time not being able to effectively zoom and pan on the data. It's not really a problem of dealing with huge data sets, it's just viewing them effectively.

author: Harvs
author_rank: Super Contributor
timestamp: September 25, 2013, 01:05:22 pm
content: Quote from: David_AVD on September 25, 2013, 09:49:08 amYeah just graphs. All time domain.

author: Harvs
author_rank: Super Contributor
timestamp: September 25, 2013, 01:07:50 pm
content: Quote from: johnwa on September 25, 2013, 11:20:06 amI like this idea, perhaps not for the main data analysis, but I need to provide a web accessable front end as well. How processing heavy is Gnuplot? Could it produce a PNG in a reasonable time on a RPi?

author: edpalmer42
author_rank: Super Contributor
timestamp: September 25, 2013, 03:22:05 pm
content: Although this software has some specialized features that might not be useful to you, it has the basic importing, plotting, zooming, etc. features that you described. Also, it's free!http://ulrich-bangert.de/Plotter.zipI've attached a screen capture showing a plot of frequency vs. time with a one second increment.Ed

author: dfmischler
author_rank: Frequent Contributor
timestamp: September 25, 2013, 03:49:56 pm
content: I have a bunch of temperature, humidity, power, etc. info. I started with Excel, then wrote some code to feed it to gnuplot.

author: Rigby
author_rank: Super Contributor
timestamp: September 26, 2013, 12:29:50 pm
content: Perl was pretty much designed for this. It is very good at filtering huge amounts of text down to small amounts of text, and just text manipulation in general. (It's good at a lot of other things, too.)Someone mentioned GNUPlot, which I would agree with. It is not newbie-friendly, but it can graph just about anything. You can pretty much feed GNUPlot your entire, unaltered data file, and if you're clever about what you tell it to do, you'll get very meaningful graphs out of it.If you're looking for some pre-made solution that you don't want to spend a lot of time on, then I don't know of anything off the shelf. Every place I've worked that had this requirement wound up writing their own manipulation layer with GNUPlot as the output layer.Best of luck!

author: ivaylo
author_rank: Frequent Contributor
timestamp: September 27, 2013, 07:26:43 am
content: The R package -http://www.r-project.organd Octave -http://www.gnu.org/software/octave/are free and do visualizations out of the box.If you want to serve your graphs and view in a browser you can do:-https://developers.google.com/chart/- may be the easiest of them all, zoom and pan may be a bit harder-http://processingjs.org- really cool, easy to make interactive- someone mentioned D3 but it ain't for the faint of heart, IMOAll of these require programming and designing your UI and controls though. And if you want some of it done for you, you can use something likehttp://www.highcharts.comwhich they say is free for non commercial use (still have to cook up something serving the Ajax it needs I believe).

author: AndreasF
author_rank: Frequent Contributor
timestamp: September 27, 2013, 05:58:10 pm
content: If you have access to Matlab, then Matlab, otherwise the open-source alternative Octave should work as well.

author: bingo600
author_rank: Super Contributor
timestamp: September 27, 2013, 06:30:19 pm
content: Although this is free linux programs , many are also available for Win.http://www.linuxlinks.com/article/20080803104017665/Scientific.htmlhttp://linuxmint.tumblr.com/post/27284846595/92-of-the-best-free-scientific-software-for-linuxmatplotlib Python based plotting system in a style similar to Matlab (Win exe's avaiable)http://matplotlib.org//Bingo

author: TMM
author_rank: Frequent Contributor
timestamp: September 27, 2013, 06:50:40 pm
content: Matlab can get pretty slow once you start graphing a few meg points. Zooming becomes very tedious at that stage as it stops responding for long periods while it regraphs.Even if that doesn't bother you, it will alias like no tomorrow so you probably won't be able to find a transient event within several meg points of data just by visual inspection.You are best off reducing the amount of samples (peak detect) before graphing which you could do with Excel anyway, or just write a script which identifies the points of interest.

author: Smokey
author_rank: Super Contributor
timestamp: September 27, 2013, 09:31:45 pm
content: Do you really have to scan over the plotted data with your eyeballs, or can you write an algorithm to go through the data dumps and pick out the interesting bits for you and present only those?Python is really good at that kind of stuff with lots of libraries for analysis and display.

author: johnwa
author_rank: Frequent Contributor
timestamp: September 28, 2013, 10:17:42 am
content: Quote from: Harvs on September 25, 2013, 01:07:50 pmI don't think it is too bad speed-wise. I don't really know how much horsepower the RPi has, but graphing a week's worth of data (~10000 samples x 4 channels) only takes a second or two on an old 1000MHz-class laptop (of which a significant proportion will be overheads in the network and HTTP server)You can also apply smoothing to the traces using Gnuplot, though I have found this gets a little messy when you are viewing multiple datasets simultaneously.

author: kaz911
author_rank: Super Contributor
timestamp: September 28, 2013, 10:22:37 am
content: Try Qlikview - really quick for table based dataIt is expensive though - but I use it often as excel replacement for quick data overview with great search. Linked tables works really well too.

author: eliocor
author_rank: Unknown
timestamp: September 28, 2013, 11:58:37 am
content: I useDPlotand I am very happy: it is able to handle millions of points without any hitch and is very fast.You can download a 30 day trial version, so you can check if it can suit your needs.

author: ivaylo
author_rank: Frequent Contributor
timestamp: September 29, 2013, 07:41:16 am
content: QuoteYes, forgot about qlikview. Very powerful and has a free version with no cap on the size of the data set AFAIK.